[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:01 dist-test-slave-mwkb krb5kdc[1038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:01 dist-test-slave-mwkb krb5kdc[1038](info): set up 2 sockets
Jun 20 19:17:01 dist-test-slave-mwkb krb5kdc[1038](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:17:02.621443 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:02 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:02.726073 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38693
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:02.799408 1051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:02.799633 1051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:02.799686 1051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:02.800869 1051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:02.800911 1051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:02.800926 1051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:02.800938 1051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:02.800999 1051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:02.802407 1051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:02.802575 1051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:02.802817 1051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:02.807931 1051 server_base.cc:1034] running on GCE node
W20260620 19:17:02.808014 1056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:02.807901 1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:02.807919 1057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:02.808355 1051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:02.808678 1051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:02.809834 1051 hybrid_clock.cc:648] HybridClock initialized: now 1781983022809813 us; error 59 us; skew 500 ppm
Jun 20 19:17:02 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983022, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:02.812155 1051 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:02.812608 1051 webserver.cc:460] Webserver started at http://127.1.1.254:35791/ using document root <none> and password file <none>
I20260620 19:17:02.812842 1051 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:02.812909 1051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:02.813061 1051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:02.814049 1051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "7a77357928bf4138b569603e424fc279"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "09db7dfe0b7f22b32a5fc9611336d45e"
server_key_iv: "aa4aca0e7c0dbecfa448eb9961c8c5de"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.814404 1051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "7a77357928bf4138b569603e424fc279"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "09db7dfe0b7f22b32a5fc9611336d45e"
server_key_iv: "aa4aca0e7c0dbecfa448eb9961c8c5de"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.815855 1051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:02.816551 1066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:02.816741 1051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:02.816860 1051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "7a77357928bf4138b569603e424fc279"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "09db7dfe0b7f22b32a5fc9611336d45e"
server_key_iv: "aa4aca0e7c0dbecfa448eb9961c8c5de"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.816933 1051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:02.826997 1051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:02.828992 1051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:02.829167 1051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:02.833050 1051 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:38693
I20260620 19:17:02.833093 1118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:38693 every 8 connection(s)
I20260620 19:17:02.833425 1051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:02.834136 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:02.835897 1119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279: Bootstrap starting.
I20260620 19:17:02.836354 1119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:02.836560 1119 log.cc:824] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:02.837173 1119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279: No bootstrap required, opened a new log
I20260620 19:17:02.838516 1119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } }
I20260620 19:17:02.838645 1119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:02.838665 1119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a77357928bf4138b569603e424fc279, State: Initialized, Role: FOLLOWER
I20260620 19:17:02.838794 1119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } }
I20260620 19:17:02.838871 1119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:02.838892 1119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:02.838917 1119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:02.839319 1119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } }
I20260620 19:17:02.839392 1119 leader_election.cc:302] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a77357928bf4138b569603e424fc279; no voters:
I20260620 19:17:02.839494 1119 leader_election.cc:288] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:02.839563 1122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:02.839798 1119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:02.839974 1122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [term 1 LEADER]: Becoming Leader. State: Replica: 7a77357928bf4138b569603e424fc279, State: Running, Role: LEADER
I20260620 19:17:02.840008 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1051
I20260620 19:17:02.840073 1122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } }
I20260620 19:17:02.840106 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:02.840397 1122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a77357928bf4138b569603e424fc279. Latest consensus state: current_term: 1 leader_uuid: "7a77357928bf4138b569603e424fc279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } } }
I20260620 19:17:02.840508 1122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:02.840418 1123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a77357928bf4138b569603e424fc279" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77357928bf4138b569603e424fc279" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38693 } } }
I20260620 19:17:02.840569 1123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:02.840862 1031 external_mini_cluster.cc:1428] Setting key 23f157d4215508990075e34b391cfe74
I20260620 19:17:02.841571 1131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:02.842047 1131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:02.844666 1131 catalog_manager.cc:1347] Generated new cluster ID: 41d3c1250580408bae61233355321d52
I20260620 19:17:02.844728 1131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:17:02 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:02.854527 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:02.841861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.1.1.254:38693)
0620 19:17:02.842054 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:02.842060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:02.842089 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:17:02.844229 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:02.844234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:02.844288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:02.844416 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:02.845975 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:02.846811 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:02.847782 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:02.848139 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:02.848259 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:02.851211 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:02.851217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:02.851217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:02.851234 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:02.852780 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:02.853290 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:02.853293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:02.853293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:02.853374 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:02.853623 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:02.853624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:02.853624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:02.853835 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:02.854001 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:02.854229 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 19:17:02.854348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
I20260620 19:17:02.864745 1131 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:02.865243 1131 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:02.868477 1131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a77357928bf4138b569603e424fc279: Generated new TSK 0
I20260620 19:17:02.868616 1131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:02.897588 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:02.980010 1145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:02.980185 1145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:02.980208 1145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:02.981410 1145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:02.981448 1145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:02.981501 1145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:02.982853 1145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:02.983078 1145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:02.983282 1145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:02.987948 1153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:02.988042 1151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:02.987948 1150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:02.988155 1145 server_base.cc:1034] running on GCE node
I20260620 19:17:02.988298 1145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:02.988461 1145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:02.989603 1145 hybrid_clock.cc:648] HybridClock initialized: now 1781983022989593 us; error 23 us; skew 500 ppm
Jun 20 19:17:02 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983022, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:02.991925 1145 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:02.992370 1145 webserver.cc:460] Webserver started at http://127.1.1.193:36639/ using document root <none> and password file <none>
I20260620 19:17:02.992590 1145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:02.992632 1145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:02.992786 1145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:02.993781 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "906de1e16902442f9d4b3aefb34fde52"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "287902f8f62a97435e7485c6e1cbcd2c"
server_key_iv: "1bae6e4cd2f6ecccac956371de6954c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.994108 1145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "906de1e16902442f9d4b3aefb34fde52"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "287902f8f62a97435e7485c6e1cbcd2c"
server_key_iv: "1bae6e4cd2f6ecccac956371de6954c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.995405 1145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:02.996057 1160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:02.996243 1145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:02.996308 1145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "906de1e16902442f9d4b3aefb34fde52"
format_stamp: "Formatted at 2026-06-20 19:17:02 on dist-test-slave-mwkb"
server_key: "287902f8f62a97435e7485c6e1cbcd2c"
server_key_iv: "1bae6e4cd2f6ecccac956371de6954c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:02.996390 1145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:03.014575 1145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:03.016144 1145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:03.016307 1145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:03.016512 1145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:03.016803 1145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:03.016834 1145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.016855 1145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:03.016889 1145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.023211 1145 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:37671
I20260620 19:17:03.023257 1273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:37671 every 8 connection(s)
I20260620 19:17:03.023551 1145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983022, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:03.031889 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.024141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:48215 (local address 127.1.1.254:38693)
0620 19:17:03.024181 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:03.024182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:03.024400 (+ 218us) server_negotiation.cc:407] Connection header received
0620 19:17:03.024862 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:03.024863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:03.024875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:03.024915 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:03.025867 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.026346 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.027269 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.027462 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.027540 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.029382 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:03.029383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:03.029384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:03.029392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:03.030814 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.031177 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.031177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.031178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.031222 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.031426 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.031427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.031427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.031514 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:03.031558 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:03.031765 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:17:03.031833 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:03.031994 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.024136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38693 (local address 127.1.1.193:48215)
0620 19:17:03.024293 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:03.024323 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:03.024754 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:03.024975 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:03.024979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:03.025589 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:03.025761 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.025765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.026436 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.026438 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.027168 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.027170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.027549 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.027550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.027672 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.028031 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:03.028039 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:03.029296 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:03.030892 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.030898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.030899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.031094 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.031291 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.031291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.031292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.031374 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.031566 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:03.031570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:03.031667 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 19:17:03.031736 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:17:03.031835 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 19:17:03.032069 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1145
I20260620 19:17:03.032141 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:03.032262 1031 external_mini_cluster.cc:1428] Setting key 025328d2dc00bd69745eafeccbe1e706
I20260620 19:17:03.032505 1274 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38693
I20260620 19:17:03.032612 1274 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:03.032768 1274 heartbeater.cc:511] Master 127.1.1.254:38693 requested a full tablet report, sending...
I20260620 19:17:03.033272 1083 ts_manager.cc:194] Registered new tserver with Master: 906de1e16902442f9d4b3aefb34fde52 (127.1.1.193:37671)
I20260620 19:17:03.033869 1083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:48215
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:03.053823 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:03.125985 1279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:03.126137 1279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:03.126154 1279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:03.127120 1279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:03.127156 1279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:03.127177 1279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:03.128353 1279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:03.128530 1279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:03.128670 1279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:03.133103 1285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:03.133167 1279 server_base.cc:1034] running on GCE node
W20260620 19:17:03.133138 1284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:03.133134 1287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:03.133452 1279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:03.133656 1279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:03.134819 1279 hybrid_clock.cc:648] HybridClock initialized: now 1781983023134813 us; error 29 us; skew 500 ppm
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:03.136907 1279 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:03.137352 1279 webserver.cc:460] Webserver started at http://127.1.1.194:42737/ using document root <none> and password file <none>
I20260620 19:17:03.137558 1279 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:03.137619 1279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:03.137737 1279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:03.138597 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "3eab2a7d77c949aea57e6dc7ed44db0a"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "8dd7507d6b4f4dc362ee8e2d18a88764"
server_key_iv: "c0d238d89d819d90719ce20ffb7c71b9"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.138922 1279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "3eab2a7d77c949aea57e6dc7ed44db0a"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "8dd7507d6b4f4dc362ee8e2d18a88764"
server_key_iv: "c0d238d89d819d90719ce20ffb7c71b9"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.140220 1279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:03.140880 1294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.141165 1279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:03.141273 1279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "3eab2a7d77c949aea57e6dc7ed44db0a"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "8dd7507d6b4f4dc362ee8e2d18a88764"
server_key_iv: "c0d238d89d819d90719ce20ffb7c71b9"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.141353 1279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:03.164477 1279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:03.165963 1279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:03.166102 1279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:03.166285 1279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:03.166550 1279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:03.166599 1279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.166628 1279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:03.166651 1279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.172456 1279 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:43627
I20260620 19:17:03.172487 1407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:43627 every 8 connection(s)
I20260620 19:17:03.172749 1279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:03.177747 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1279
I20260620 19:17:03.177840 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:03.177978 1031 external_mini_cluster.cc:1428] Setting key a7fd7a57416567e948c4a4073282ad4e
I20260620 19:17:03.181809 1410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.173340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38693 (local address 127.1.1.194:46189)
0620 19:17:03.173472 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:03.173506 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:03.173895 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:03.174068 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:03.174070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:03.174542 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:03.174695 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.174699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.175249 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.175250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.175937 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.175938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.176224 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.176224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.176312 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.176747 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:03.176761 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:03.177953 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:03.180063 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.180069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.180070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.180333 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.180899 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.180900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.180900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.181036 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.181303 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:03.181308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:03.181466 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 19:17:03.181559 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:17:03.181663 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260620 19:17:03.181844 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.173386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:46189 (local address 127.1.1.254:38693)
0620 19:17:03.173423 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:03.173425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:03.173579 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:17:03.173965 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:03.173966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:03.173974 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:03.174018 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:03.174770 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.175184 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.176000 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.176162 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.176206 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.178038 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:03.178040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:03.178040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:03.178048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:03.179984 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.180759 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.180760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.180761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.180827 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.181136 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.181137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.181137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.181237 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:03.181289 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:03.181606 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 19:17:03.181689 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:03.182451 1408 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38693
I20260620 19:17:03.182540 1408 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:03.182689 1408 heartbeater.cc:511] Master 127.1.1.254:38693 requested a full tablet report, sending...
I20260620 19:17:03.183055 1083 ts_manager.cc:194] Registered new tserver with Master: 3eab2a7d77c949aea57e6dc7ed44db0a (127.1.1.194:43627)
I20260620 19:17:03.183581 1083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:46189
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:03.201383 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:03.272855 1413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:03.273010 1413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:03.273029 1413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:03.274070 1413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:03.274120 1413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:03.274168 1413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:03.275414 1413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:03.275620 1413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:03.275815 1413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:03.280057 1419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:03.280194 1421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:03.280264 1418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:03.280699 1413 server_base.cc:1034] running on GCE node
I20260620 19:17:03.280915 1413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:03.281203 1413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:03.282361 1413 hybrid_clock.cc:648] HybridClock initialized: now 1781983023282348 us; error 28 us; skew 500 ppm
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:03.284556 1413 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:03.284855 1413 webserver.cc:460] Webserver started at http://127.1.1.195:38703/ using document root <none> and password file <none>
I20260620 19:17:03.285120 1413 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:03.285173 1413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:03.285293 1413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:03.286209 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "c3a5a0a9feeb4c058443617922134f86"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "c20d4513b3043fe86605aed76ae6278b"
server_key_iv: "f5f17eeaeb35e0a52dddb824f6a70d81"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.286543 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "c3a5a0a9feeb4c058443617922134f86"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "c20d4513b3043fe86605aed76ae6278b"
server_key_iv: "f5f17eeaeb35e0a52dddb824f6a70d81"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.287712 1413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:03.288347 1428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.288530 1413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:03.288594 1413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "c3a5a0a9feeb4c058443617922134f86"
format_stamp: "Formatted at 2026-06-20 19:17:03 on dist-test-slave-mwkb"
server_key: "c20d4513b3043fe86605aed76ae6278b"
server_key_iv: "f5f17eeaeb35e0a52dddb824f6a70d81"
server_key_version: "encryptionkey@0"
I20260620 19:17:03.288671 1413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:03.302464 1413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:03.304287 1413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:03.304420 1413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:03.304601 1413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:03.304867 1413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:03.304895 1413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.304912 1413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:03.304958 1413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:03.310424 1413 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:36447
I20260620 19:17:03.310497 1541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:36447 every 8 connection(s)
I20260620 19:17:03.310786 1413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:03.314855 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1413
I20260620 19:17:03.314927 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:03.315073 1031 external_mini_cluster.cc:1428] Setting key e8276f39992e15c24c2f84fd40cc0da1
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:03.318321 1140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.311461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:59377 (local address 127.1.1.254:38693)
0620 19:17:03.311526 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:03.311527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:03.311653 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:17:03.312000 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:03.312001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:03.312009 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:03.312054 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:03.312713 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.313143 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.313960 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.314132 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.314176 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.316135 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:03.316137 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:03.316137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:03.316144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:03.317434 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.317721 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.317722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.317722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.317763 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.317939 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.317940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.317940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.318020 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:03.318062 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:03.318224 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:17:03.318269 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:17:03.318502 1544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.311435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38693 (local address 127.1.1.195:59377)
0620 19:17:03.311553 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:03.311585 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:03.311934 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:03.312091 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:03.312093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:03.312513 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:03.312652 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.312655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.313214 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.313215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.313891 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:03.313893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.314197 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.314198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:03.314281 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.314617 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:03.314623 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:03.316058 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:03.317493 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.317497 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.317498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.317665 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.317819 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:03.317819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:03.317819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:03.317893 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:03.318066 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:03.318069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:03.318150 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 19:17:03.318213 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:17:03.318304 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260620 19:17:03.318907 1542 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38693
I20260620 19:17:03.319000 1542 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:03.319147 1542 heartbeater.cc:511] Master 127.1.1.254:38693 requested a full tablet report, sending...
I20260620 19:17:03.319442 1083 ts_manager.cc:194] Registered new tserver with Master: c3a5a0a9feeb4c058443617922134f86 (127.1.1.195:36447)
I20260620 19:17:03.319832 1083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:59377
I20260620 19:17:03.326984 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:03.347508 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.340517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.1.1.193:37671)
0620 19:17:03.340645 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:03.340670 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:03.340681 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:03.340728 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:03.340730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:03.340747 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:03.340813 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:03.341527 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.342173 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.342883 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.343153 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.343230 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.345189 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:03.345196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:03.345196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:03.345206 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:03.346379 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.346708 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.346711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.346711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.346761 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.346951 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.346951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.346952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.347066 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:03.347210 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:03.347305 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:17:03.347379 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260620 19:17:03.347833 1188 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42410
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:03.366228 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:03.360533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.1.1.193:37671)
0620 19:17:03.360591 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:03.360593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:03.360693 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:17:03.360768 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:03.360769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:03.360781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:03.360844 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:03.361419 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.361819 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.362427 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:03.362593 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:03.362666 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:03.364420 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:03.364421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:03.364421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:03.364426 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:03.365483 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.365764 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.365765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.365765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.365807 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:03.365938 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:03.365939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:03.365939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:03.366003 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:03.366074 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:03.366133 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:17:03.366181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:17:03.366647 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1145
I20260620 19:17:03.371085 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1279
I20260620 19:17:03.376619 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1413
I20260620 19:17:03.380550 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1051
2026-06-20T19:17:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1727 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1556](info): set up 2 sockets
Jun 20 19:17:03 dist-test-slave-mwkb krb5kdc[1556](info): commencing operation
krb5kdc: starting...
W20260620 19:17:04.332301 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:04.433857 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:43791
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:04.509006 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:04.509194 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:04.509215 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:04.510248 1569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:04.510285 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:04.510299 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:04.510311 1569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:04.510324 1569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:04.511658 1569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:04.511822 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:04.511963 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:04.516613 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.516662 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.516625 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:04.516656 1569 server_base.cc:1034] running on GCE node
I20260620 19:17:04.517040 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:04.517328 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:04.518466 1569 hybrid_clock.cc:648] HybridClock initialized: now 1781983024518454 us; error 27 us; skew 500 ppm
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:04.524147 1569 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:04.524529 1569 webserver.cc:460] Webserver started at http://127.1.1.254:34405/ using document root <none> and password file <none>
I20260620 19:17:04.524758 1569 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:04.524832 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:04.524984 1569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:04.525990 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "3180e4c9ce314d7aab758155d5741795"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "3b41d0743435d46eaee566a21fa167df"
server_key_iv: "a20940a4b2c96687dafc0b250bcbfd0c"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.526333 1569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "3180e4c9ce314d7aab758155d5741795"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "3b41d0743435d46eaee566a21fa167df"
server_key_iv: "a20940a4b2c96687dafc0b250bcbfd0c"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.527609 1569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:04.528313 1584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.528502 1569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:04.528584 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "3180e4c9ce314d7aab758155d5741795"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "3b41d0743435d46eaee566a21fa167df"
server_key_iv: "a20940a4b2c96687dafc0b250bcbfd0c"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.528635 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:04.539755 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:04.541926 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:04.542037 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:04.545542 1569 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:43791
I20260620 19:17:04.545588 1636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:43791 every 8 connection(s)
I20260620 19:17:04.545979 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:04.546833 1637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:04.547487 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1569
I20260620 19:17:04.547576 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:04.547747 1031 external_mini_cluster.cc:1428] Setting key 116bfa5e1e1ffe4484cf4c88358b4df5
I20260620 19:17:04.549803 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795: Bootstrap starting.
I20260620 19:17:04.550451 1637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:04.550734 1637 log.cc:824] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:04.551403 1637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795: No bootstrap required, opened a new log
I20260620 19:17:04.553516 1637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } }
I20260620 19:17:04.553633 1637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:04.553654 1637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3180e4c9ce314d7aab758155d5741795, State: Initialized, Role: FOLLOWER
I20260620 19:17:04.553783 1637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } }
I20260620 19:17:04.553896 1637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:04.553936 1637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:04.553987 1637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:04.554508 1637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } }
I20260620 19:17:04.554658 1637 leader_election.cc:302] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3180e4c9ce314d7aab758155d5741795; no voters:
I20260620 19:17:04.554853 1637 leader_election.cc:288] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:04.554879 1642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:04.555002 1642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [term 1 LEADER]: Becoming Leader. State: Replica: 3180e4c9ce314d7aab758155d5741795, State: Running, Role: LEADER
I20260620 19:17:04.555096 1637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:04.555076 1642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } }
I20260620 19:17:04.555426 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3180e4c9ce314d7aab758155d5741795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } } }
I20260620 19:17:04.555510 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:04.555642 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3180e4c9ce314d7aab758155d5741795. Latest consensus state: current_term: 1 leader_uuid: "3180e4c9ce314d7aab758155d5741795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3180e4c9ce314d7aab758155d5741795" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43791 } } }
I20260620 19:17:04.555712 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:04.556072 1648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:04.556617 1648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:04.558332 1648 catalog_manager.cc:1347] Generated new cluster ID: 4ebb109dd8ce4cfcb2a8775a2382ad24
I20260620 19:17:04.558382 1648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:04.558805 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:04.548252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.1.1.254:43791)
0620 19:17:04.548402 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:04.548410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:04.548440 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:17:04.548918 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:04.548926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:04.548979 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:04.549139 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:04.549676 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.551027 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.551822 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.552399 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.552516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:04.554850 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:04.554854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:04.554854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:04.554865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:04.556901 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.557398 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.557400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.557400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.557491 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.557714 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.557714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.557715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.557933 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:04.558440 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:04.558502 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:17:04.558610 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 19:17:04.582327 1648 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:04.582993 1648 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:04.588771 1648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3180e4c9ce314d7aab758155d5741795: Generated new TSK 0
I20260620 19:17:04.588932 1648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:04.611485 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:04.689337 1663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:04.689518 1663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:04.689549 1663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:04.690658 1663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:04.690728 1663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:04.690775 1663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:04.692237 1663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:04.692474 1663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:04.692698 1663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:04.697072 1663 server_base.cc:1034] running on GCE node
W20260620 19:17:04.697057 1669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.697175 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.697062 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:04.697553 1663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:04.697742 1663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:04.698879 1663 hybrid_clock.cc:648] HybridClock initialized: now 1781983024698857 us; error 36 us; skew 500 ppm
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:04.701165 1663 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:04.701582 1663 webserver.cc:460] Webserver started at http://127.1.1.193:34261/ using document root <none> and password file <none>
I20260620 19:17:04.701792 1663 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:04.701854 1663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:04.701967 1663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:04.702857 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "4f4864d65b1842619c4bc2c19725025a"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "2beea38d3de9a716c3f0c70702cd6f5b"
server_key_iv: "d188cf1a129be512f257c0624564a888"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.703195 1663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "4f4864d65b1842619c4bc2c19725025a"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "2beea38d3de9a716c3f0c70702cd6f5b"
server_key_iv: "d188cf1a129be512f257c0624564a888"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.704476 1663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:04.705178 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.705384 1663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.705476 1663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "4f4864d65b1842619c4bc2c19725025a"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "2beea38d3de9a716c3f0c70702cd6f5b"
server_key_iv: "d188cf1a129be512f257c0624564a888"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.705560 1663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:04.719936 1663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:04.721719 1663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:04.721866 1663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:04.722054 1663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:04.722313 1663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:04.722363 1663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.722393 1663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:04.722417 1663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.727885 1663 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:43687
I20260620 19:17:04.727931 1791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:43687 every 8 connection(s)
I20260620 19:17:04.728202 1663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:04.735729 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1663
I20260620 19:17:04.735847 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:04.735997 1031 external_mini_cluster.cc:1428] Setting key 01c489a717c38d3ce9daed2d28e74571
I20260620 19:17:04.736495 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:04.728845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:46323 (local address 127.1.1.254:43791)
0620 19:17:04.728880 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:04.728882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:04.729049 (+ 167us) server_negotiation.cc:407] Connection header received
0620 19:17:04.729865 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:04.729866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:04.729874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:04.729923 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:04.730440 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.730904 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.731897 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.732132 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.732180 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:04.734119 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:04.734121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:04.734121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:04.734127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:04.735250 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.735560 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.735561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.735561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.735622 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.735871 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.735871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.735872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.736021 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:04.736090 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:04.736340 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:17:04.736414 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:17:04.736617 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:04.728802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43791 (local address 127.1.1.193:46323)
0620 19:17:04.728939 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:04.728972 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:04.729782 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:04.729967 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:04.729970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:04.730208 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:04.730369 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:04.730373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.731021 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.731022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:04.731826 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:04.731828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.732276 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.732277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:04.732389 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:04.732833 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:04.732840 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:04.734031 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:04.735314 (+ 1283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:04.735320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:04.735321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:04.735499 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:04.735696 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:04.735697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:04.735697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:04.735797 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:04.736092 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:04.736099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:04.736212 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 19:17:04.736306 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:17:04.736432 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260620 19:17:04.736994 1792 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43791
I20260620 19:17:04.737118 1792 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:04.737264 1792 heartbeater.cc:511] Master 127.1.1.254:43791 requested a full tablet report, sending...
I20260620 19:17:04.737707 1601 ts_manager.cc:194] Registered new tserver with Master: 4f4864d65b1842619c4bc2c19725025a (127.1.1.193:43687)
I20260620 19:17:04.738541 1601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:46323
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:04.757884 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:04.829808 1797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:04.830040 1797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:04.830108 1797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:04.831338 1797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:04.831419 1797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:04.831460 1797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:04.833525 1797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:04.833778 1797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:04.834000 1797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:04.838682 1805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.838687 1802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.838687 1803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:04.838888 1797 server_base.cc:1034] running on GCE node
I20260620 19:17:04.839097 1797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:04.839339 1797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:04.840495 1797 hybrid_clock.cc:648] HybridClock initialized: now 1781983024840467 us; error 45 us; skew 500 ppm
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:04.842881 1797 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:04.843305 1797 webserver.cc:460] Webserver started at http://127.1.1.194:39055/ using document root <none> and password file <none>
I20260620 19:17:04.843505 1797 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:04.843560 1797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:04.843734 1797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:04.844707 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "53c0926735eb4492ab5a1c477ccf278f"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "f00c8f1d1bc9cabc5dd94e9b4caf0c3c"
server_key_iv: "639546e20da5ae993d9b527a3946fc2b"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.845057 1797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "53c0926735eb4492ab5a1c477ccf278f"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "f00c8f1d1bc9cabc5dd94e9b4caf0c3c"
server_key_iv: "639546e20da5ae993d9b527a3946fc2b"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.846227 1797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:04.846882 1812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.847075 1797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.847158 1797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "53c0926735eb4492ab5a1c477ccf278f"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "f00c8f1d1bc9cabc5dd94e9b4caf0c3c"
server_key_iv: "639546e20da5ae993d9b527a3946fc2b"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.847242 1797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:04.860406 1797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:04.862567 1797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:04.862725 1797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:04.862941 1797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:04.863212 1797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:04.863243 1797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.863263 1797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:04.863312 1797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.868707 1797 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:44587
I20260620 19:17:04.868767 1925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:44587 every 8 connection(s)
I20260620 19:17:04.869045 1797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:04.871618 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1797
I20260620 19:17:04.871706 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:04.871860 1031 external_mini_cluster.cc:1428] Setting key da26a53731e3e09677f364b166852616
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:04.878829 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:04.869758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:43125 (local address 127.1.1.254:43791)
0620 19:17:04.869799 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:04.869801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:04.869962 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:17:04.870849 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:04.870850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:04.870859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:04.870908 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:04.871478 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.872273 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.873448 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.873722 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.873793 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:04.875948 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:04.875950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:04.875951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:04.875959 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:04.877435 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.877861 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.877863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.877863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.877924 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:04.878198 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:04.878199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:04.878199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:04.878318 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:04.878400 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:04.878657 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 19:17:04.878736 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:04.879014 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:04.869745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43791 (local address 127.1.1.194:43125)
0620 19:17:04.869871 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:04.869901 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:04.870759 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:04.870957 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:04.870961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:04.871187 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:04.871359 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:04.871362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.872378 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.872379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:04.873350 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:04.873352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:04.873802 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:04.873803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:04.873921 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:04.874436 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:04.874444 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:04.875850 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:04.877540 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:04.877546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:04.877548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:04.877769 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:04.877998 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:04.877999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:04.877999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:04.878105 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:04.878391 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:04.878397 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:04.878529 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 19:17:04.878636 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:17:04.878792 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
I20260620 19:17:04.879441 1926 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43791
I20260620 19:17:04.879529 1926 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:04.879746 1926 heartbeater.cc:511] Master 127.1.1.254:43791 requested a full tablet report, sending...
I20260620 19:17:04.880111 1601 ts_manager.cc:194] Registered new tserver with Master: 53c0926735eb4492ab5a1c477ccf278f (127.1.1.194:44587)
I20260620 19:17:04.880517 1601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:43125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:04.895718 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:04.971994 1931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:04.972147 1931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:04.972167 1931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:04.973062 1931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:04.973134 1931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:04.973158 1931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:04.974630 1931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43791
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:04.974792 1931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:04.974942 1931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:04.979357 1939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:04.979385 1937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:04.979743 1931 server_base.cc:1034] running on GCE node
W20260620 19:17:04.979802 1936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:04.980115 1931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:04.980392 1931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:04.981559 1931 hybrid_clock.cc:648] HybridClock initialized: now 1781983024981538 us; error 35 us; skew 500 ppm
Jun 20 19:17:04 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:04.983896 1931 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:04.984236 1931 webserver.cc:460] Webserver started at http://127.1.1.195:45807/ using document root <none> and password file <none>
I20260620 19:17:04.984404 1931 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:04.984440 1931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:04.984517 1931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:04.985473 1931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "634d53812eda44b79c0ebf2789d4ddfe"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "a1512634f6162eda4c240006db85ee45"
server_key_iv: "58ca18f77ad27bd47dc3bee08f469352"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.985733 1931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "634d53812eda44b79c0ebf2789d4ddfe"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "a1512634f6162eda4c240006db85ee45"
server_key_iv: "58ca18f77ad27bd47dc3bee08f469352"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.986833 1931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:04.987411 1946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:04.987586 1931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:04.987684 1931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "634d53812eda44b79c0ebf2789d4ddfe"
format_stamp: "Formatted at 2026-06-20 19:17:04 on dist-test-slave-mwkb"
server_key: "a1512634f6162eda4c240006db85ee45"
server_key_iv: "58ca18f77ad27bd47dc3bee08f469352"
server_key_version: "encryptionkey@0"
I20260620 19:17:04.987766 1931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:04.999465 1931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:05.001572 1931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:05.001713 1931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:05.001899 1931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:05.002152 1931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:05.002202 1931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:05.002238 1931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:05.002261 1931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:05.007277 1931 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:38853
I20260620 19:17:05.007323 2059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:38853 every 8 connection(s)
I20260620 19:17:05.007596 1931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:05.009618 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 1931
I20260620 19:17:05.009732 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:05.009912 1031 external_mini_cluster.cc:1428] Setting key 8b7b0c1edc3c04f0660e2a2cf1afc46f
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:05.016103 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.008303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:56081 (local address 127.1.1.254:43791)
0620 19:17:05.008348 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.008351 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.008543 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:17:05.009527 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.009529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.009539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.009599 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:05.010340 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.011009 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.011891 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.012049 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.012092 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.014050 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:05.014051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:05.014051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:05.014056 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:05.015134 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.015444 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.015445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.015446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.015489 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.015668 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.015668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.015669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.015758 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:05.015803 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.015993 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.016049 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:05.016232 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.008237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43791 (local address 127.1.1.195:56081)
0620 19:17:05.008410 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.008448 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.009422 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.009661 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.009664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.009987 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:05.010245 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.010251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.011079 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.011080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.011825 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.011827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.012110 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.012111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.012197 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.012680 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:05.012689 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:05.013981 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:05.015200 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:05.015205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:05.015207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:05.015386 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:05.015541 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:05.015541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:05.015542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:05.015621 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:05.015805 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:05.015808 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:05.015898 (+ 90us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.015966 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.016063 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":71,"threads_started":1}
I20260620 19:17:05.016633 2060 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43791
I20260620 19:17:05.016721 2060 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:05.016860 2060 heartbeater.cc:511] Master 127.1.1.254:43791 requested a full tablet report, sending...
I20260620 19:17:05.017192 1601 ts_manager.cc:194] Registered new tserver with Master: 634d53812eda44b79c0ebf2789d4ddfe (127.1.1.195:38853)
I20260620 19:17:05.017537 1601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:56081
I20260620 19:17:05.021529 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:05.027596 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.022908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.1.1.254:43791)
0620 19:17:05.022965 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.022967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.023042 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:17:05.023072 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.023073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.023079 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.023122 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:05.023575 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.023963 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.024562 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.024708 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.024757 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.025774 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:05.025774 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:05.025775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:05.025780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:05.026808 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.027074 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.027075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.027075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.027114 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.027213 (+ 99us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.027213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.027214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.027339 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:05.027396 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.027480 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.027524 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:17:05.029575 1601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:05.030995 1601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:05.036827 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.032910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:38853 (local address 127.0.0.1:37594)
0620 19:17:05.033054 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.033101 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.033156 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.033535 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.033536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.033538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:05.033589 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.033591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034537 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.034539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.035744 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.035746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036525 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.036597 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.036608 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.036650 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.036696 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260620 19:17:05.036849 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.033050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.1.1.194:44587)
0620 19:17:05.033262 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.033266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.033318 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:17:05.033403 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.033404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.033415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.033492 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:05.033625 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034662 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.035750 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036402 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036487 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.036551 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.036735 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.036779 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260620 19:17:05.037005 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.033163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.1.1.193:43687)
0620 19:17:05.033307 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.033311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.033669 (+ 358us) server_negotiation.cc:407] Connection header received
0620 19:17:05.033731 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.033733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.033746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.033832 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:05.033938 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034899 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036001 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036542 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036602 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.036654 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.036873 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.036919 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260620 19:17:05.037156 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.032928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:43687 (local address 127.0.0.1:54102)
0620 19:17:05.033615 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.033624 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.033671 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.033845 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.033846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.033848 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:05.033882 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.033883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034999 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.035000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.035929 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.035930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036685 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.036753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.036756 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.036802 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.036837 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":53,"threads_started":1}
I20260620 19:17:05.036842 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.032917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:44587 (local address 127.0.0.1:36192)
0620 19:17:05.033265 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.033275 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.033328 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.033516 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.033519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.033521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:05.033573 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.033574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034778 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.034779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.035667 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.035669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036576 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.036577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.036621 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.036623 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.036665 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.036697 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":22,"thread_start_us":52,"threads_started":1}
I20260620 19:17:05.037272 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.033141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.1.1.195:38853)
0620 19:17:05.033415 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.033417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.033423 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:05.033445 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.033446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.033455 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.033535 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:05.033663 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.034437 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.035806 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.036399 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.037028 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.037124 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.037148 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.037192 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":62,"threads_started":1}
I20260620 19:17:05.037760 1726 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd6546afb8749c386d3035b9675ff7f (DEFAULT_TABLE table=test-table [id=669e9627799047d999007985bfb68baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:05.037851 1860 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd6546afb8749c386d3035b9675ff7f (DEFAULT_TABLE table=test-table [id=669e9627799047d999007985bfb68baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:05.037963 1994 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd6546afb8749c386d3035b9675ff7f (DEFAULT_TABLE table=test-table [id=669e9627799047d999007985bfb68baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:05.038076 1726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd6546afb8749c386d3035b9675ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:05.038084 1860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd6546afb8749c386d3035b9675ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:05.038260 1994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd6546afb8749c386d3035b9675ff7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:05.040531 2074 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Bootstrap starting.
I20260620 19:17:05.041103 2076 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Bootstrap starting.
I20260620 19:17:05.041327 2075 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Bootstrap starting.
I20260620 19:17:05.041517 2074 tablet_bootstrap.cc:654] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:05.041700 2076 tablet_bootstrap.cc:654] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:05.041879 2075 tablet_bootstrap.cc:654] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:05.042039 2074 log.cc:824] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:05.042217 2076 log.cc:824] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:05.042433 2075 log.cc:824] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:05.042812 2074 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: No bootstrap required, opened a new log
I20260620 19:17:05.042894 2074 ts_tablet_manager.cc:1397] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:05.043062 2075 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: No bootstrap required, opened a new log
I20260620 19:17:05.043110 2075 ts_tablet_manager.cc:1397] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:05.044462 2074 raft_consensus.cc:348] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.044615 2074 raft_consensus.cc:374] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:05.044659 2074 raft_consensus.cc:729] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634d53812eda44b79c0ebf2789d4ddfe, State: Initialized, Role: FOLLOWER
I20260620 19:17:05.044788 2074 consensus_queue.cc:260] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.044895 2075 raft_consensus.cc:348] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.045056 2075 raft_consensus.cc:374] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:05.045095 2074 ts_tablet_manager.cc:1428] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:17:05.045112 2060 heartbeater.cc:503] Master 127.1.1.254:43791 was elected leader, sending a full tablet report...
I20260620 19:17:05.045123 2075 raft_consensus.cc:729] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4864d65b1842619c4bc2c19725025a, State: Initialized, Role: FOLLOWER
I20260620 19:17:05.045236 2075 consensus_queue.cc:260] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.045495 1792 heartbeater.cc:503] Master 127.1.1.254:43791 was elected leader, sending a full tablet report...
I20260620 19:17:05.045536 2076 tablet_bootstrap.cc:492] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: No bootstrap required, opened a new log
I20260620 19:17:05.045482 2075 ts_tablet_manager.cc:1428] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:17:05.045583 2076 ts_tablet_manager.cc:1397] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:17:05.046941 2076 raft_consensus.cc:348] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.047125 2076 raft_consensus.cc:374] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:05.047168 2076 raft_consensus.cc:729] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c0926735eb4492ab5a1c477ccf278f, State: Initialized, Role: FOLLOWER
I20260620 19:17:05.047274 2076 consensus_queue.cc:260] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.047526 2076 ts_tablet_manager.cc:1428] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:05.047551 1926 heartbeater.cc:503] Master 127.1.1.254:43791 was elected leader, sending a full tablet report...
W20260620 19:17:05.119730 1927 tablet.cc:2369] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:05.228981 1793 tablet.cc:2369] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:05.258103 2061 tablet.cc:2369] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:17:05.286303 2081 raft_consensus.cc:482] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:05.286425 2081 raft_consensus.cc:504] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.286768 2081 leader_election.cc:288] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634d53812eda44b79c0ebf2789d4ddfe (127.1.1.195:38853), 53c0926735eb4492ab5a1c477ccf278f (127.1.1.194:44587)
I20260620 19:17:05.289381 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.286900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:38853 (local address 127.1.1.193:52893)
0620 19:17:05.287014 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.287026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.287098 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.287243 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.287244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.287246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:05.287327 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.287329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.287936 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.287938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.288659 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.288661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.289167 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.289169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.289241 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.289245 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.289296 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.289323 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":19,"thread_start_us":54,"threads_started":1}
I20260620 19:17:05.289426 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.286948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52893 (local address 127.1.1.195:38853)
0620 19:17:05.287006 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.287008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.287099 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:17:05.287159 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.287160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.287173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.287217 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:05.287382 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.287858 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.288751 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.289104 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.289151 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.289158 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.289309 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.289342 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:17:05.289529 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.286931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:44587 (local address 127.1.1.193:35623)
0620 19:17:05.287058 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:05.287091 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:05.287132 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:05.287285 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:05.287285 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:05.287287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:05.287384 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.287386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.287985 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.287986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.288668 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:05.288670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.289268 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.289269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:05.289359 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.289363 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:05.289416 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:17:05.289454 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":18,"thread_start_us":62,"threads_started":1}
I20260620 19:17:05.289562 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.287059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:35623 (local address 127.1.1.194:44587)
0620 19:17:05.287111 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.287113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.287132 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:17:05.287189 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.287190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.287218 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.287262 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:05.287474 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.287874 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.288785 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.289179 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.289241 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.289249 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.289445 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.289486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:17:05.289703 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd6546afb8749c386d3035b9675ff7f" candidate_uuid: "4f4864d65b1842619c4bc2c19725025a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634d53812eda44b79c0ebf2789d4ddfe" is_pre_election: true
I20260620 19:17:05.289783 1880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd6546afb8749c386d3035b9675ff7f" candidate_uuid: "4f4864d65b1842619c4bc2c19725025a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c0926735eb4492ab5a1c477ccf278f" is_pre_election: true
I20260620 19:17:05.289852 2014 raft_consensus.cc:2454] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f4864d65b1842619c4bc2c19725025a in term 0.
I20260620 19:17:05.289891 1880 raft_consensus.cc:2454] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f4864d65b1842619c4bc2c19725025a in term 0.
I20260620 19:17:05.290082 1679 leader_election.cc:302] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4864d65b1842619c4bc2c19725025a, 634d53812eda44b79c0ebf2789d4ddfe; no voters:
I20260620 19:17:05.290249 2081 raft_consensus.cc:2781] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:05.290303 2081 raft_consensus.cc:482] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:05.290333 2081 raft_consensus.cc:3037] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:05.290973 2081 raft_consensus.cc:504] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.291103 2081 leader_election.cc:288] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [CANDIDATE]: Term 1 election: Requested vote from peers 634d53812eda44b79c0ebf2789d4ddfe (127.1.1.195:38853), 53c0926735eb4492ab5a1c477ccf278f (127.1.1.194:44587)
I20260620 19:17:05.291263 2014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd6546afb8749c386d3035b9675ff7f" candidate_uuid: "4f4864d65b1842619c4bc2c19725025a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634d53812eda44b79c0ebf2789d4ddfe"
I20260620 19:17:05.291317 1880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd6546afb8749c386d3035b9675ff7f" candidate_uuid: "4f4864d65b1842619c4bc2c19725025a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c0926735eb4492ab5a1c477ccf278f"
I20260620 19:17:05.291347 2014 raft_consensus.cc:3037] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:05.291376 1880 raft_consensus.cc:3037] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:05.292110 2014 raft_consensus.cc:2454] T 0dd6546afb8749c386d3035b9675ff7f P 634d53812eda44b79c0ebf2789d4ddfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f4864d65b1842619c4bc2c19725025a in term 1.
I20260620 19:17:05.292110 1880 raft_consensus.cc:2454] T 0dd6546afb8749c386d3035b9675ff7f P 53c0926735eb4492ab5a1c477ccf278f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f4864d65b1842619c4bc2c19725025a in term 1.
I20260620 19:17:05.292255 1679 leader_election.cc:302] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f4864d65b1842619c4bc2c19725025a, 634d53812eda44b79c0ebf2789d4ddfe; no voters:
I20260620 19:17:05.292344 2081 raft_consensus.cc:2781] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:05.292464 2081 raft_consensus.cc:686] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [term 1 LEADER]: Becoming Leader. State: Replica: 4f4864d65b1842619c4bc2c19725025a, State: Running, Role: LEADER
I20260620 19:17:05.292557 2081 consensus_queue.cc:237] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } }
I20260620 19:17:05.293319 1601 catalog_manager.cc:5495] T 0dd6546afb8749c386d3035b9675ff7f P 4f4864d65b1842619c4bc2c19725025a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f4864d65b1842619c4bc2c19725025a (127.1.1.193). New cstate: current_term: 1 leader_uuid: "4f4864d65b1842619c4bc2c19725025a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634d53812eda44b79c0ebf2789d4ddfe" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 38853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c0926735eb4492ab5a1c477ccf278f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 44587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f4864d65b1842619c4bc2c19725025a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43687 } health_report { overall_health: HEALTHY } } }
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:05.322764 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.316550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.1.1.193:43687)
0620 19:17:05.316594 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.316596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.316698 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:17:05.316735 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.316735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.316743 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.316785 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:05.317225 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.317600 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.318166 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.318331 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.318373 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.320517 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:05.320522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:05.320522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:05.320527 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:05.321855 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.322183 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.322186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.322187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.322250 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.322391 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.322391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.322392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.322489 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:05.322599 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.322648 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.322704 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983025, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:05.342288 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.336386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.1.1.193:43687)
0620 19:17:05.336454 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.336456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.336521 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:17:05.336581 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.336582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.336590 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.336626 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:05.337059 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.337450 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.338062 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.338222 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.338292 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.340103 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:05.340105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:05.340105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:05.340112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:05.341282 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.341585 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.341586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.341586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.341635 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.341824 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.341824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.341825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.341907 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:05.342015 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.342141 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.342201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[1556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:05.361766 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:05.355533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.1.1.193:43687)
0620 19:17:05.355595 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:05.355597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:05.355683 (+ 86us) server_negotiation.cc:407] Connection header received
0620 19:17:05.355779 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:05.355780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:05.355789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:05.355866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:05.356376 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.356767 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.357492 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:05.357648 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:05.357724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:05.359463 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:05.359464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:05.359464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:05.359470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:05.360799 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.361133 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.361134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.361135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.361181 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:05.361390 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:05.361391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:05.361391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:05.361469 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:05.361546 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:05.361619 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:17:05.361687 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 19:17:05.362150 1706 tablet_service.cc:2908] Rejecting scan request for tablet 0dd6546afb8749c386d3035b9675ff7f: Uninitialized: clean time has not yet been initialized
I20260620 19:17:05.362926 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1663
I20260620 19:17:05.367449 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1797
I20260620 19:17:05.371670 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1931
I20260620 19:17:05.376171 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 1569
2026-06-20T19:17:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1996 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[2099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[2099](info): set up 2 sockets
Jun 20 19:17:05 dist-test-slave-mwkb krb5kdc[2099](info): commencing operation
krb5kdc: starting...
W20260620 19:17:06.360333 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:06.457814 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34605
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:34605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:06.530071 2112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:06.530273 2112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:06.530295 2112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:06.531314 2112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:06.531349 2112 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:17:06.531363 2112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:06.531388 2112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:06.531400 2112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:06.531412 2112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:06.532552 2112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:34605
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:06.532708 2112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:06.532836 2112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:06.537053 2112 server_base.cc:1034] running on GCE node
W20260620 19:17:06.536983 2117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.536983 2118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.536993 2120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.537434 2112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:06.537690 2112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:06.538842 2112 hybrid_clock.cc:648] HybridClock initialized: now 1781983026538824 us; error 32 us; skew 500 ppm
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:06.541036 2112 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:06.541486 2112 webserver.cc:460] Webserver started at http://127.1.1.254:40415/ using document root <none> and password file <none>
I20260620 19:17:06.541685 2112 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:06.541742 2112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:06.541858 2112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:06.542701 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "254712098978105ea6512567b058930e"
server_key_iv: "f9c030c8538b04f8f5cabca1dc20098d"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.543018 2112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "254712098978105ea6512567b058930e"
server_key_iv: "f9c030c8538b04f8f5cabca1dc20098d"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.544116 2112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:06.544659 2127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.544788 2112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.544865 2112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "254712098978105ea6512567b058930e"
server_key_iv: "f9c030c8538b04f8f5cabca1dc20098d"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.544932 2112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:06.566471 2112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:06.568301 2112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:06.568435 2112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:06.572819 2112 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:34605
I20260620 19:17:06.572860 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:34605 every 8 connection(s)
I20260620 19:17:06.573182 2112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:06.573722 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:06.575673 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Bootstrap starting.
I20260620 19:17:06.576200 2190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:06.576455 2190 log.cc:824] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:06.577232 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: No bootstrap required, opened a new log
I20260620 19:17:06.579123 2190 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } }
I20260620 19:17:06.579263 2190 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:06.579305 2190 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ac7080b0c342eaa6ebe9f9490dabfd, State: Initialized, Role: FOLLOWER
I20260620 19:17:06.579438 2190 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } }
I20260620 19:17:06.579511 2190 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:06.579553 2190 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:06.579610 2190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:06.580117 2190 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } }
I20260620 19:17:06.580235 2190 leader_election.cc:302] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7ac7080b0c342eaa6ebe9f9490dabfd; no voters:
I20260620 19:17:06.580386 2190 leader_election.cc:288] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:06.580425 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:06.580603 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [term 1 LEADER]: Becoming Leader. State: Replica: c7ac7080b0c342eaa6ebe9f9490dabfd, State: Running, Role: LEADER
I20260620 19:17:06.580647 2190 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:06.580745 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } }
I20260620 19:17:06.581099 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } } }
I20260620 19:17:06.581107 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7ac7080b0c342eaa6ebe9f9490dabfd. Latest consensus state: current_term: 1 leader_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7ac7080b0c342eaa6ebe9f9490dabfd" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34605 } } }
I20260620 19:17:06.581233 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:06.581441 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:06.581463 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2112
I20260620 19:17:06.581553 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:06.581557 2201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:06.581710 1031 external_mini_cluster.cc:1428] Setting key 0f6d3823a3523a748c7b0f4d9a72b924
I20260620 19:17:06.582015 2201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:06.583436 2201 catalog_manager.cc:1347] Generated new cluster ID: d58c626aac004a90bd4af088b39ec634
I20260620 19:17:06.583485 2201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:06.590776 2201 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:06.591462 2201 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:06.592768 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.582228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.1.1.254:34605)
0620 19:17:06.582854 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:06.582862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:06.582888 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:17:06.583029 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:06.583036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:06.583095 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:06.583227 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:06.583882 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.585174 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.586081 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.586717 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.586782 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.589419 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:06.589422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:06.589423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:06.589433 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:06.591274 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.591659 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.591662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.591662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.591744 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.591903 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.591904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.591904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.592125 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:06.592236 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:06.592438 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:17:06.592561 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":52,"threads_started":1}
I20260620 19:17:06.596170 2201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Generated new TSK 0
I20260620 19:17:06.596314 2201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:06.621871 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34605
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:06.696944 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:06.697156 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:06.697183 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:06.698210 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:06.698243 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:06.698278 2216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:06.698309 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:06.699476 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:06.699697 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:06.699891 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:06.704309 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.704316 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.704370 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.704336 2216 server_base.cc:1034] running on GCE node
I20260620 19:17:06.704632 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:06.704867 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:06.706007 2216 hybrid_clock.cc:648] HybridClock initialized: now 1781983026706002 us; error 31 us; skew 500 ppm
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:06.707979 2216 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:06.708349 2216 webserver.cc:460] Webserver started at http://127.1.1.193:43899/ using document root <none> and password file <none>
I20260620 19:17:06.708525 2216 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:06.708561 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:06.708695 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:06.709611 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "8a92ef627f564481b2a94eb00eb3859d"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "ecaa76aee17e84f3cf9ffeddc3fa858f"
server_key_iv: "73dc290cc4dce7e5556639dd6f499165"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.709933 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "8a92ef627f564481b2a94eb00eb3859d"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "ecaa76aee17e84f3cf9ffeddc3fa858f"
server_key_iv: "73dc290cc4dce7e5556639dd6f499165"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.711107 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:06.711731 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.711879 2216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:06.711946 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "8a92ef627f564481b2a94eb00eb3859d"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "ecaa76aee17e84f3cf9ffeddc3fa858f"
server_key_iv: "73dc290cc4dce7e5556639dd6f499165"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.711990 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:06.728255 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:06.730016 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:06.730168 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:06.730674 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:06.730715 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.730739 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:06.730783 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.736527 2216 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:36709
I20260620 19:17:06.736598 2346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:36709 every 8 connection(s)
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:06.736840 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:06.739490 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.731132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:37657 (local address 127.1.1.254:34605)
0620 19:17:06.731208 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:06.731211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:06.731368 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:17:06.731810 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:06.731811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:06.731824 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:06.731900 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:06.732629 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.733367 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.734662 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.734968 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.735042 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.737272 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:06.737274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:06.737274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:06.737279 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:06.738498 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.738786 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.738787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.738787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.738829 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.738994 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.738994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.738994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.739073 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:06.739113 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:06.739358 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 19:17:06.739419 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:17:06.739676 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.731090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.193:37657)
0620 19:17:06.731244 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:06.731282 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:06.731718 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:06.731958 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:06.731975 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:06.732272 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:06.732520 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.732526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.733461 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.733461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.734577 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.734580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.735057 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.735058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.735177 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.735592 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:06.735600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:06.737207 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:06.738555 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.738558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.738559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.738732 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.738875 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.738875 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.738875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.738954 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.739121 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:06.739125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:06.739255 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 19:17:06.739333 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:17:06.739438 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 19:17:06.740155 2347 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34605
I20260620 19:17:06.740238 2347 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:06.740396 2347 heartbeater.cc:511] Master 127.1.1.254:34605 requested a full tablet report, sending...
I20260620 19:17:06.740891 2143 ts_manager.cc:194] Registered new tserver with Master: 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709)
I20260620 19:17:06.741829 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:37657
I20260620 19:17:06.745751 2353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.741086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.193:52125)
0620 19:17:06.741203 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:06.741213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:06.741267 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:06.741386 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:06.741386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:06.741553 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:06.741602 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.741603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.742240 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.742241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.742771 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.742773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.743090 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.743091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.743165 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.743460 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:06.743463 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:06.743689 (+ 226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:06.744830 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.744830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.744831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.744994 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.745197 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.745198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.745198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.745248 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.745459 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:06.745460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:06.745497 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:17:06.745575 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:17:06.745661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260620 19:17:06.745733 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.741132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52125 (local address 127.1.1.254:34605)
0620 19:17:06.741165 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:06.741167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:06.741258 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:17:06.741320 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:06.741321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:06.741327 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:06.741382 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:06.741662 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.742157 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.742856 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.743028 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.743090 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.743797 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:06.743798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:06.743799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:06.743805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:06.744760 (+ 955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.745086 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.745087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.745087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.745134 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.745309 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.745310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.745310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.745403 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:06.745465 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:06.745597 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:17:06.745659 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:06.745950 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2216
I20260620 19:17:06.746045 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:06.746210 1031 external_mini_cluster.cc:1428] Setting key c6805c84cb54aed9e5b5d4f7e9d0afa5
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:06.767513 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34605
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:06.840162 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:06.840380 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:06.840406 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:06.841590 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:06.841631 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:06.841650 2356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:06.841665 2356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:06.842834 2356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:06.842988 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:06.843124 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:06.847956 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.847988 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.848173 2356 server_base.cc:1034] running on GCE node
W20260620 19:17:06.847956 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.848475 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:06.848712 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:06.849871 2356 hybrid_clock.cc:648] HybridClock initialized: now 1781983026849858 us; error 40 us; skew 500 ppm
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:06.852185 2356 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:06.852614 2356 webserver.cc:460] Webserver started at http://127.1.1.194:46101/ using document root <none> and password file <none>
I20260620 19:17:06.852829 2356 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:06.852891 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:06.853024 2356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:06.853945 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "192902ec9a894645ae0e1a7015f5027a"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "c82a22a67906fa3dee3c869394769a55"
server_key_iv: "c9469042fe05d636ed3169b34815f745"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.854252 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "192902ec9a894645ae0e1a7015f5027a"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "c82a22a67906fa3dee3c869394769a55"
server_key_iv: "c9469042fe05d636ed3169b34815f745"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.855413 2356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:06.856041 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.856180 2356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:06.856266 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "192902ec9a894645ae0e1a7015f5027a"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "c82a22a67906fa3dee3c869394769a55"
server_key_iv: "c9469042fe05d636ed3169b34815f745"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.856331 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:06.863040 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:06.864389 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:06.864511 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:06.864880 2356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:06.864933 2356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.864967 2356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:06.865003 2356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:06.871126 2356 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:41577
I20260620 19:17:06.871183 2486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:41577 every 8 connection(s)
I20260620 19:17:06.871470 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:06.873615 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.865245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:51945 (local address 127.1.1.254:34605)
0620 19:17:06.865308 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:06.865310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:06.865483 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:17:06.865825 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:06.865827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:06.865840 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:06.865902 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:06.866557 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.867281 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.868586 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.868939 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.869013 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.871190 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:06.871192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:06.871192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:06.871200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:06.872562 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.872896 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.872897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.872897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.872944 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.873145 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.873145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.873146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.873247 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:06.873294 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:06.873496 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:17:06.873558 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:17:06.873759 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.865223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.194:51945)
0620 19:17:06.865378 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:06.865408 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:06.865745 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:06.865958 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:06.865961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:06.866229 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:06.866451 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.866456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.867361 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.867362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.868500 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.868503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.869030 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.869030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.869191 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.869658 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:06.869670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:06.871104 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:06.872624 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.872629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.872630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.872835 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.872982 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.872982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.872982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.873055 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.873296 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:06.873300 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:06.873396 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:17:06.873471 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:17:06.873568 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260620 19:17:06.874167 2487 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34605
I20260620 19:17:06.874241 2487 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:06.874388 2487 heartbeater.cc:511] Master 127.1.1.254:34605 requested a full tablet report, sending...
I20260620 19:17:06.874744 2143 ts_manager.cc:194] Registered new tserver with Master: 192902ec9a894645ae0e1a7015f5027a (127.1.1.194:41577)
I20260620 19:17:06.875103 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:51945
I20260620 19:17:06.879828 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.875138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:55733 (local address 127.1.1.254:34605)
0620 19:17:06.875170 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:06.875172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:06.875268 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:17:06.875331 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:06.875331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:06.875340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:06.875383 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:06.875670 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.876118 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.876812 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.877002 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.877096 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.877918 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:06.877919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:06.877919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:06.877925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:06.878992 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.879269 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.879269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.879269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.879321 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:06.879453 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:06.879454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:06.879454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:06.879566 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:06.879621 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:06.879720 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:17:06.879772 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:06.879834 2493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:06.875097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.194:55733)
0620 19:17:06.875218 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:06.875227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:06.875287 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:06.875385 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:06.875385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:06.875547 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:06.875596 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.875597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.876204 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.876205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.876717 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:06.876719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:06.877094 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:06.877095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:06.877167 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:06.877571 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:06.877576 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:06.877849 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:06.879055 (+ 1206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.879055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.879056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.879212 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.879371 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:06.879372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:06.879372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:06.879418 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:06.879619 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:06.879620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:06.879655 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:17:06.879704 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:17:06.879770 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260620 19:17:06.881500 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2356
I20260620 19:17:06.881575 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:06.881729 1031 external_mini_cluster.cc:1428] Setting key e200088c532cd017c416acb9be5cb07f
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:06.903966 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34605
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:06.979846 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:06.980053 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:06.980136 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:06.981362 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:06.981424 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:06.981457 2496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:06.981501 2496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:06.982826 2496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:34605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:06.983055 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:06.983254 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:06.988019 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.988121 2496 server_base.cc:1034] running on GCE node
W20260620 19:17:06.987972 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:06.987977 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:06.988438 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:06.988662 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:06.989806 2496 hybrid_clock.cc:648] HybridClock initialized: now 1781983026989790 us; error 32 us; skew 500 ppm
Jun 20 19:17:06 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:06.991957 2496 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:06.992343 2496 webserver.cc:460] Webserver started at http://127.1.1.195:39801/ using document root <none> and password file <none>
I20260620 19:17:06.992548 2496 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:06.992604 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:06.992726 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:06.993672 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "f3f102d3d20147949c8f9997a41d50a8"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "f4f1f98e2967592d9b2ca06d995fee9a"
server_key_iv: "11e5835c2033dac17ab2f03b6b92a37e"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.994001 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "f3f102d3d20147949c8f9997a41d50a8"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "f4f1f98e2967592d9b2ca06d995fee9a"
server_key_iv: "11e5835c2033dac17ab2f03b6b92a37e"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.995182 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:06.995857 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:06.996013 2496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:06.996098 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "f3f102d3d20147949c8f9997a41d50a8"
format_stamp: "Formatted at 2026-06-20 19:17:06 on dist-test-slave-mwkb"
server_key: "f4f1f98e2967592d9b2ca06d995fee9a"
server_key_iv: "11e5835c2033dac17ab2f03b6b92a37e"
server_key_version: "encryptionkey@0"
I20260620 19:17:06.996166 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:07.002221 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:07.003633 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:07.003770 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:07.004204 2496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:07.004252 2496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.004285 2496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:07.004307 2496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:07.011021 2496 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:36703
I20260620 19:17:07.011319 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:07.011432 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:36703 every 8 connection(s)
I20260620 19:17:07.013126 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.004501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:57861 (local address 127.1.1.254:34605)
0620 19:17:07.004558 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.004561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.004751 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:17:07.005231 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.005233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.005249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.005319 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.005838 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.006664 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.007545 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.007850 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.007921 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.010609 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.010610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.010611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.010618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.012081 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.012394 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.012395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.012395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.012439 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.012628 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.012628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.012629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.012720 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.012765 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.012968 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.013027 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:17:07.013231 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.004495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.195:57861)
0620 19:17:07.004640 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.004672 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.005040 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.005369 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.005373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.005575 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:07.005736 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.005739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.006752 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.006753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.007461 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.007463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.007945 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.007946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.008068 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.008491 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:07.008507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:07.010536 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:07.012140 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.012143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.012145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.012328 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.012487 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.012488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.012488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.012564 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.012767 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:07.012770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:07.012861 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.012932 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.013036 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260620 19:17:07.013602 2627 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34605
I20260620 19:17:07.013672 2627 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:07.013842 2627 heartbeater.cc:511] Master 127.1.1.254:34605 requested a full tablet report, sending...
I20260620 19:17:07.014127 2143 ts_manager.cc:194] Registered new tserver with Master: f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703)
I20260620 19:17:07.014547 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:57861
I20260620 19:17:07.017997 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2496
I20260620 19:17:07.018100 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:07.018273 1031 external_mini_cluster.cc:1428] Setting key dedbd3a4034d7307b1068a47b375c4b0
I20260620 19:17:07.018976 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:07.019737 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.014512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:46443 (local address 127.1.1.254:34605)
0620 19:17:07.014544 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.014546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.014632 (+ 86us) server_negotiation.cc:407] Connection header received
0620 19:17:07.014706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.014707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.014716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.014764 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.015042 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.015530 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.016179 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.016369 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.016443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.017243 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.017244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.017244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.017250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.018735 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.019039 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.019040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.019041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.019092 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.019258 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.019259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.019259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.019356 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.019424 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.019559 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.019635 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:07.019763 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.014471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.1.1.195:46443)
0620 19:17:07.014566 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.014574 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.014652 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.014766 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.014767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.014913 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:07.014954 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.014955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.015610 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.015611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.016092 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.016093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.016445 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.016446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.016513 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.016883 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:07.016887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:07.017154 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:07.018804 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.018805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.018806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.018960 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.019138 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.019139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.019140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.019193 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.019417 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:07.019417 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:07.019456 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.019534 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.019636 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":57,"threads_started":1}
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:07.048537 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.041034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.1.1.254:34605)
0620 19:17:07.041099 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.041101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.041108 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:07.041129 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.041130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.041140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.041212 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.041790 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.042575 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.043430 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.043620 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.043686 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.046414 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.046415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.046415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.046421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.047540 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.047823 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.047824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.047824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.047889 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.048048 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.048049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.048049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.048153 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.048263 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.048380 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.048445 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:07.049878 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:07.051388 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:07.057011 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:41577 (local address 127.0.0.1:39032)
0620 19:17:07.053385 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.053427 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.053489 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.053917 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.053918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.053920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.053997 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.053999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.055230 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.055231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.055863 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.055864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056713 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.056802 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.056807 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.056838 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.056878 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":5,"thread_start_us":50,"threads_started":1}
I20260620 19:17:07.057099 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.1.1.193:36709)
0620 19:17:07.053789 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.053792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.053800 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:07.053832 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.053836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.053848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.053941 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.054039 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.055113 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056218 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056747 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056815 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.056866 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.056934 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.056976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":65,"threads_started":1}
I20260620 19:17:07.057106 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36709 (local address 127.0.0.1:48648)
0620 19:17:07.053599 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.053606 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.053640 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.053950 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.053951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.053952 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.053984 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.053985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.055194 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.055195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.056140 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.056142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056819 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.056871 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.056873 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.056924 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.056956 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260620 19:17:07.056967 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36703 (local address 127.0.0.1:56628)
0620 19:17:07.053565 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.053575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.053617 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.053882 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.053885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.053887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.053953 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.053955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.054831 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.054831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.055843 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.055846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056633 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.056705 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.056719 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.056804 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.056854 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":5,"thread_start_us":48,"threads_started":1}
I20260620 19:17:07.057477 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.1.1.195:36703)
0620 19:17:07.053753 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.053756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.053763 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:07.053787 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.053788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.053798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.053875 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.054028 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.054757 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.055924 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056535 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056623 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.056682 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.057333 (+ 651us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.057377 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260620 19:17:07.057605 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.053370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.1.1.194:41577)
0620 19:17:07.053740 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.053745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.053756 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:07.053793 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.053796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.053812 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.054109 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.054171 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.055138 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.055980 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.056623 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.056998 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.057433 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.057459 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.057504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":83,"threads_started":1}
I20260620 19:17:07.057929 2281 tablet_service.cc:1467] Processing CreateTablet for tablet c383f21df8794410860e5fd078e3979e (DEFAULT_TABLE table=test-table [id=701f0d981ca34fea927a15a74a791e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.057963 2561 tablet_service.cc:1467] Processing CreateTablet for tablet c383f21df8794410860e5fd078e3979e (DEFAULT_TABLE table=test-table [id=701f0d981ca34fea927a15a74a791e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.058179 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c383f21df8794410860e5fd078e3979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.058200 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c383f21df8794410860e5fd078e3979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.060545 2647 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Bootstrap starting.
I20260620 19:17:07.061188 2648 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Bootstrap starting.
I20260620 19:17:07.061443 2647 tablet_bootstrap.cc:654] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.061699 2648 tablet_bootstrap.cc:654] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.061743 2647 log.cc:824] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:07.061782 2421 tablet_service.cc:1467] Processing CreateTablet for tablet c383f21df8794410860e5fd078e3979e (DEFAULT_TABLE table=test-table [id=701f0d981ca34fea927a15a74a791e22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.061946 2648 log.cc:824] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:07.062036 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c383f21df8794410860e5fd078e3979e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.062433 2647 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: No bootstrap required, opened a new log
I20260620 19:17:07.062489 2647 ts_tablet_manager.cc:1397] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:17:07.064317 2647 raft_consensus.cc:348] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.064467 2647 raft_consensus.cc:374] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.064504 2647 raft_consensus.cc:729] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f102d3d20147949c8f9997a41d50a8, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.064648 2647 consensus_queue.cc:260] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.064881 2647 ts_tablet_manager.cc:1428] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:17:07.064906 2627 heartbeater.cc:503] Master 127.1.1.254:34605 was elected leader, sending a full tablet report...
I20260620 19:17:07.065155 2648 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: No bootstrap required, opened a new log
I20260620 19:17:07.065218 2648 ts_tablet_manager.cc:1397] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:17:07.065810 2652 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Bootstrap starting.
I20260620 19:17:07.066347 2652 tablet_bootstrap.cc:654] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.066633 2652 log.cc:824] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:07.066968 2648 raft_consensus.cc:348] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.067150 2648 raft_consensus.cc:374] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.067183 2648 raft_consensus.cc:729] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a92ef627f564481b2a94eb00eb3859d, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.067305 2652 tablet_bootstrap.cc:492] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: No bootstrap required, opened a new log
I20260620 19:17:07.067287 2648 consensus_queue.cc:260] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.067363 2652 ts_tablet_manager.cc:1397] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:07.067525 2648 ts_tablet_manager.cc:1428] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:07.067551 2347 heartbeater.cc:503] Master 127.1.1.254:34605 was elected leader, sending a full tablet report...
I20260620 19:17:07.069037 2652 raft_consensus.cc:348] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.069196 2652 raft_consensus.cc:374] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.069226 2652 raft_consensus.cc:729] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192902ec9a894645ae0e1a7015f5027a, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.069293 2652 consensus_queue.cc:260] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.069471 2652 ts_tablet_manager.cc:1428] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:07.069516 2487 heartbeater.cc:503] Master 127.1.1.254:34605 was elected leader, sending a full tablet report...
W20260620 19:17:07.121927 2488 tablet.cc:2369] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:17:07.228813 2655 raft_consensus.cc:482] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:07.228960 2655 raft_consensus.cc:504] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.229307 2655 leader_election.cc:288] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703), 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709)
I20260620 19:17:07.232126 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.229537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:55569 (local address 127.1.1.195:36703)
0620 19:17:07.229585 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.229587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.229597 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:17:07.229623 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.229624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.229635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.229696 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.229889 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.230414 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231424 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.231758 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.231853 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.232023 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.232062 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:17:07.232173 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.229464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36703 (local address 127.1.1.194:55569)
0620 19:17:07.229511 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.229522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.229560 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.229720 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.229721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.229722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.229803 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.229805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.230483 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.230484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.231319 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.231322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.231848 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.231946 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.231949 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.232016 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.232053 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":60}
I20260620 19:17:07.232183 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.229569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36709 (local address 127.1.1.194:36527)
0620 19:17:07.229718 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.229728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.229781 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.229896 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.229896 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.229898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.229966 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.229967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.230572 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.230573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.231314 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.231316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.231916 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.231994 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.231997 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.232044 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.232085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260620 19:17:07.232419 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.229670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:36527 (local address 127.1.1.193:36709)
0620 19:17:07.229711 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.229713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.229781 (+ 68us) server_negotiation.cc:407] Connection header received
0620 19:17:07.229820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.229821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.229832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.229881 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.230022 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.230497 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231397 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.231806 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.231897 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.231906 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.232119 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.232159 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:07.232524 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c383f21df8794410860e5fd078e3979e" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f102d3d20147949c8f9997a41d50a8" is_pre_election: true
I20260620 19:17:07.232664 2581 raft_consensus.cc:2454] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 0.
I20260620 19:17:07.232739 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c383f21df8794410860e5fd078e3979e" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d" is_pre_election: true
I20260620 19:17:07.232893 2301 raft_consensus.cc:2454] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 0.
I20260620 19:17:07.232895 2373 leader_election.cc:302] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192902ec9a894645ae0e1a7015f5027a, f3f102d3d20147949c8f9997a41d50a8; no voters:
I20260620 19:17:07.233002 2655 raft_consensus.cc:2781] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:07.233054 2655 raft_consensus.cc:482] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:07.233099 2655 raft_consensus.cc:3037] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.234115 2655 raft_consensus.cc:504] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.234229 2655 leader_election.cc:288] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 election: Requested vote from peers f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703), 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709)
I20260620 19:17:07.234409 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c383f21df8794410860e5fd078e3979e" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d"
I20260620 19:17:07.234480 2301 raft_consensus.cc:3037] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.235179 2301 raft_consensus.cc:2454] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 1.
I20260620 19:17:07.235309 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c383f21df8794410860e5fd078e3979e" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f102d3d20147949c8f9997a41d50a8"
I20260620 19:17:07.235368 2373 leader_election.cc:302] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192902ec9a894645ae0e1a7015f5027a, 8a92ef627f564481b2a94eb00eb3859d; no voters:
I20260620 19:17:07.235394 2581 raft_consensus.cc:3037] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.235468 2655 raft_consensus.cc:2781] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:07.235598 2655 raft_consensus.cc:686] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 1 LEADER]: Becoming Leader. State: Replica: 192902ec9a894645ae0e1a7015f5027a, State: Running, Role: LEADER
I20260620 19:17:07.235695 2655 consensus_queue.cc:237] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } }
I20260620 19:17:07.236159 2581 raft_consensus.cc:2454] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 1.
I20260620 19:17:07.236608 2143 catalog_manager.cc:5495] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a reported cstate change: term changed from 0 to 1, leader changed from <none> to 192902ec9a894645ae0e1a7015f5027a (127.1.1.194). New cstate: current_term: 1 leader_uuid: "192902ec9a894645ae0e1a7015f5027a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } health_report { overall_health: UNKNOWN } } }
W20260620 19:17:07.237465 2348 tablet.cc:2369] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:07.261881 2628 tablet.cc:2369] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:17:07.282094 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.279264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.1.1.194:41577)
0620 19:17:07.279332 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.279333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.279342 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:17:07.279419 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.279420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.279434 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.279488 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:07.279590 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.280066 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.280893 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.281096 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.281150 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.281305 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.281399 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.281453 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.281507 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.281529 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:17:07.283555 2301 raft_consensus.cc:1261] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Refusing update from remote peer 192902ec9a894645ae0e1a7015f5027a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.283577 2581 raft_consensus.cc:1261] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Refusing update from remote peer 192902ec9a894645ae0e1a7015f5027a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.283851 2655 consensus_queue.cc:1035] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.283973 2657 consensus_queue.cc:1035] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.285864 2660 mvcc.cc:205] Tried to move back new op lower bound from 7299002479751221248 to 7299002479558008832. Current Snapshot: MvccSnapshot[applied={T|T < 7299002479751221248}]
I20260620 19:17:07.286720 2662 mvcc.cc:205] Tried to move back new op lower bound from 7299002479751221248 to 7299002479558008832. Current Snapshot: MvccSnapshot[applied={T|T < 7299002479751221248}]
I20260620 19:17:07.286945 2661 mvcc.cc:205] Tried to move back new op lower bound from 7299002479751221248 to 7299002479558008832. Current Snapshot: MvccSnapshot[applied={T|T < 7299002479751221248}]
I20260620 19:17:07.288825 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:07.288892 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:07.289590 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Sending DeleteTablet for 3 replicas of tablet c383f21df8794410860e5fd078e3979e
I20260620 19:17:07.289887 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet c383f21df8794410860e5fd078e3979e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39032
I20260620 19:17:07.289949 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet c383f21df8794410860e5fd078e3979e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:48648
I20260620 19:17:07.289989 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet c383f21df8794410860e5fd078e3979e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:56628
I20260620 19:17:07.290108 2670 tablet_replica.cc:330] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: stopping tablet replica
I20260620 19:17:07.290194 2670 raft_consensus.cc:2229] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:07.290163 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:17:07.290297 2670 raft_consensus.cc:2258] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:17:07.290318 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:07.290725 2670 ts_tablet_manager.cc:1905] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.291371 2671 tablet_replica.cc:330] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: stopping tablet replica
I20260620 19:17:07.291487 2671 raft_consensus.cc:2229] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:07.291509 2561 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a061d83224b44bd471ef202fac745 (DEFAULT_TABLE table=test-table [id=c53a3e2ae01e45288348e4e5ae450530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.291576 2671 raft_consensus.cc:2258] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:07.291610 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a061d83224b44bd471ef202fac745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.291714 2669 tablet_replica.cc:330] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: stopping tablet replica
I20260620 19:17:07.291788 2669 raft_consensus.cc:2229] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:07.291863 2669 raft_consensus.cc:2258] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:07.292166 2671 ts_tablet_manager.cc:1905] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.292382 2670 ts_tablet_manager.cc:1918] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:07.292435 2670 log.cc:1196] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/c383f21df8794410860e5fd078e3979e
I20260620 19:17:07.292649 2670 ts_tablet_manager.cc:1939] T c383f21df8794410860e5fd078e3979e P 8a92ef627f564481b2a94eb00eb3859d: Deleting consensus metadata
I20260620 19:17:07.292918 2647 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Bootstrap starting.
I20260620 19:17:07.293040 2129 catalog_manager.cc:4841] TS 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709): tablet c383f21df8794410860e5fd078e3979e (table test-table [id=701f0d981ca34fea927a15a74a791e22]) successfully deleted
I20260620 19:17:07.293344 2647 tablet_bootstrap.cc:654] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.293727 2671 ts_tablet_manager.cc:1918] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:07.293794 2671 log.cc:1196] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/c383f21df8794410860e5fd078e3979e
I20260620 19:17:07.293934 2647 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: No bootstrap required, opened a new log
I20260620 19:17:07.293916 2421 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a061d83224b44bd471ef202fac745 (DEFAULT_TABLE table=test-table [id=c53a3e2ae01e45288348e4e5ae450530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.293916 2281 tablet_service.cc:1467] Processing CreateTablet for tablet ed8a061d83224b44bd471ef202fac745 (DEFAULT_TABLE table=test-table [id=c53a3e2ae01e45288348e4e5ae450530]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:07.293970 2647 ts_tablet_manager.cc:1397] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:07.294010 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a061d83224b44bd471ef202fac745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.294018 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8a061d83224b44bd471ef202fac745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.294034 2671 ts_tablet_manager.cc:1939] T c383f21df8794410860e5fd078e3979e P f3f102d3d20147949c8f9997a41d50a8: Deleting consensus metadata
I20260620 19:17:07.294106 2647 raft_consensus.cc:348] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.294189 2647 raft_consensus.cc:374] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.294209 2647 raft_consensus.cc:729] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f102d3d20147949c8f9997a41d50a8, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.294247 2647 consensus_queue.cc:260] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.294322 2647 ts_tablet_manager.cc:1428] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.295121 2648 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Bootstrap starting.
I20260620 19:17:07.295178 2129 catalog_manager.cc:4841] TS f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703): tablet c383f21df8794410860e5fd078e3979e (table test-table [id=701f0d981ca34fea927a15a74a791e22]) successfully deleted
I20260620 19:17:07.295336 2652 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Bootstrap starting.
I20260620 19:17:07.295492 2648 tablet_bootstrap.cc:654] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.295732 2652 tablet_bootstrap.cc:654] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.295950 2669 ts_tablet_manager.cc:1905] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.296089 2648 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: No bootstrap required, opened a new log
I20260620 19:17:07.296144 2648 ts_tablet_manager.cc:1397] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:07.296268 2652 tablet_bootstrap.cc:492] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: No bootstrap required, opened a new log
I20260620 19:17:07.296314 2652 ts_tablet_manager.cc:1397] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:07.296428 2652 raft_consensus.cc:348] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.296489 2652 raft_consensus.cc:374] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.296507 2652 raft_consensus.cc:729] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192902ec9a894645ae0e1a7015f5027a, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.296456 2648 raft_consensus.cc:348] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.296535 2648 raft_consensus.cc:374] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.296564 2648 raft_consensus.cc:729] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a92ef627f564481b2a94eb00eb3859d, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.296542 2652 consensus_queue.cc:260] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.296599 2648 consensus_queue.cc:260] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.296617 2652 ts_tablet_manager.cc:1428] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.296674 2648 ts_tablet_manager.cc:1428] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:07.297379 2669 ts_tablet_manager.cc:1918] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:07.297438 2669 log.cc:1196] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/c383f21df8794410860e5fd078e3979e
I20260620 19:17:07.297650 2669 ts_tablet_manager.cc:1939] T c383f21df8794410860e5fd078e3979e P 192902ec9a894645ae0e1a7015f5027a: Deleting consensus metadata
I20260620 19:17:07.297957 2131 catalog_manager.cc:4841] TS 192902ec9a894645ae0e1a7015f5027a (127.1.1.194:41577): tablet c383f21df8794410860e5fd078e3979e (table test-table [id=701f0d981ca34fea927a15a74a791e22]) successfully deleted
I20260620 19:17:07.438730 2655 raft_consensus.cc:482] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:07.438810 2655 raft_consensus.cc:504] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.438910 2655 leader_election.cc:288] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709), f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703)
I20260620 19:17:07.439102 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a061d83224b44bd471ef202fac745" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d" is_pre_election: true
I20260620 19:17:07.439106 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a061d83224b44bd471ef202fac745" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f102d3d20147949c8f9997a41d50a8" is_pre_election: true
I20260620 19:17:07.439206 2301 raft_consensus.cc:2454] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 0.
I20260620 19:17:07.439204 2581 raft_consensus.cc:2454] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 0.
I20260620 19:17:07.439337 2373 leader_election.cc:302] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192902ec9a894645ae0e1a7015f5027a, 8a92ef627f564481b2a94eb00eb3859d; no voters:
I20260620 19:17:07.439427 2655 raft_consensus.cc:2781] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:07.439464 2655 raft_consensus.cc:482] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:07.439517 2655 raft_consensus.cc:3037] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.440230 2655 raft_consensus.cc:504] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.440366 2655 leader_election.cc:288] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 election: Requested vote from peers 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709), f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703)
I20260620 19:17:07.440533 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a061d83224b44bd471ef202fac745" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f102d3d20147949c8f9997a41d50a8"
I20260620 19:17:07.440613 2581 raft_consensus.cc:3037] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.441303 2581 raft_consensus.cc:2454] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 1.
I20260620 19:17:07.441453 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8a061d83224b44bd471ef202fac745" candidate_uuid: "192902ec9a894645ae0e1a7015f5027a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d"
I20260620 19:17:07.441494 2373 leader_election.cc:302] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192902ec9a894645ae0e1a7015f5027a, f3f102d3d20147949c8f9997a41d50a8; no voters:
I20260620 19:17:07.441546 2301 raft_consensus.cc:3037] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.441578 2655 raft_consensus.cc:2781] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:07.441632 2655 raft_consensus.cc:686] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 1 LEADER]: Becoming Leader. State: Replica: 192902ec9a894645ae0e1a7015f5027a, State: Running, Role: LEADER
I20260620 19:17:07.441686 2655 consensus_queue.cc:237] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.442216 2144 catalog_manager.cc:5495] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a reported cstate change: term changed from 0 to 1, leader changed from <none> to 192902ec9a894645ae0e1a7015f5027a (127.1.1.194). New cstate: current_term: 1 leader_uuid: "192902ec9a894645ae0e1a7015f5027a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:07.442281 2301 raft_consensus.cc:2454] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 192902ec9a894645ae0e1a7015f5027a in term 1.
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983026, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:07.491444 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.485565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34605 (local address 127.0.0.1:60256)
0620 19:17:07.485726 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.485733 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.485806 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.485912 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.485913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.486188 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:07.486240 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.486242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.486821 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.486821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.487380 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.487382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.487667 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.487667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.487731 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.488169 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:07.488173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:07.489364 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:07.490565 (+ 1201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.490571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.490573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.490762 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.490923 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:07.490924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:07.490924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:07.490976 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:07.491178 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:07.491184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:07.491227 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.491276 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.491357 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260620 19:17:07.491529 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.485618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.1.1.254:34605)
0620 19:17:07.485676 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.485678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.485808 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:17:07.485845 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.485846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.485854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.485902 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.486301 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.486762 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.487443 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.487611 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.487658 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.489432 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.489433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.489433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.489438 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.490501 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.490823 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.490824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.490824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.490870 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.491024 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.491024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.491024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.491130 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.491178 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.491304 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.491401 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":33}
I20260620 19:17:07.493116 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:60256:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:17:07.493304 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:07.494578 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae9d9bc8010458c87195c7a5a290cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae9ba539a8464f0e8dfaecdb3b87201b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:07.494581 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae9d9bc8010458c87195c7a5a290cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae9ba539a8464f0e8dfaecdb3b87201b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:07.494598 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae9d9bc8010458c87195c7a5a290cad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae9ba539a8464f0e8dfaecdb3b87201b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:07.494719 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae9d9bc8010458c87195c7a5a290cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.494722 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae9d9bc8010458c87195c7a5a290cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.494741 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae9d9bc8010458c87195c7a5a290cad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:07.496101 2652 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: Bootstrap starting.
I20260620 19:17:07.496095 2648 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: Bootstrap starting.
I20260620 19:17:07.496121 2647 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: Bootstrap starting.
I20260620 19:17:07.496394 2648 tablet_bootstrap.cc:654] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.496418 2652 tablet_bootstrap.cc:654] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.496481 2647 tablet_bootstrap.cc:654] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:07.496904 2652 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: No bootstrap required, opened a new log
I20260620 19:17:07.496958 2652 ts_tablet_manager.cc:1397] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:07.497120 2647 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: No bootstrap required, opened a new log
I20260620 19:17:07.497087 2652 raft_consensus.cc:348] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.497151 2652 raft_consensus.cc:374] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.497161 2647 ts_tablet_manager.cc:1397] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:07.497172 2652 raft_consensus.cc:729] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192902ec9a894645ae0e1a7015f5027a, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.497226 2652 consensus_queue.cc:260] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.497318 2652 ts_tablet_manager.cc:1428] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.497292 2647 raft_consensus.cc:348] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.497354 2647 raft_consensus.cc:374] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.497382 2647 raft_consensus.cc:729] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f102d3d20147949c8f9997a41d50a8, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.497347 2657 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.497427 2657 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.497419 2647 consensus_queue.cc:260] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.497498 2647 ts_tablet_manager.cc:1428] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.497622 2651 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.497704 2651 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.497927 2648 tablet_bootstrap.cc:492] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: No bootstrap required, opened a new log
I20260620 19:17:07.497973 2648 ts_tablet_manager.cc:1397] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:07.498099 2648 raft_consensus.cc:348] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.498172 2648 raft_consensus.cc:374] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:07.498204 2648 raft_consensus.cc:729] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a92ef627f564481b2a94eb00eb3859d, State: Initialized, Role: FOLLOWER
I20260620 19:17:07.498250 2648 consensus_queue.cc:260] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.498328 2648 ts_tablet_manager.cc:1428] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:07.498395 2654 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.498485 2654 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.498569 2651 raft_consensus.cc:482] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:07.498629 2651 raft_consensus.cc:504] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.498881 2651 leader_election.cc:288] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709), 192902ec9a894645ae0e1a7015f5027a (127.1.1.194:41577)
I20260620 19:17:07.502223 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.499015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:41483 (local address 127.1.1.193:36709)
0620 19:17:07.499049 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.499051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.499058 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:07.499135 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.499136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.499144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.499185 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.499329 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.500034 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.501407 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.501888 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.501952 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.501960 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.502138 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.502170 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:17:07.502241 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.498969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36709 (local address 127.1.1.195:41483)
0620 19:17:07.499022 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.499031 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.499068 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.499202 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.499203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.499204 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.499280 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.499282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.500102 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.500102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.501327 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.501330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.501950 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.501951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.502051 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.502054 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.502131 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.502170 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":26}
I20260620 19:17:07.502405 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae9d9bc8010458c87195c7a5a290cad" candidate_uuid: "f3f102d3d20147949c8f9997a41d50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d" is_pre_election: true
I20260620 19:17:07.502473 2301 raft_consensus.cc:2454] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f102d3d20147949c8f9997a41d50a8 in term 0.
I20260620 19:17:07.502481 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.499232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:45821 (local address 127.1.1.194:41577)
0620 19:17:07.499315 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.499317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.499324 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:07.499345 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.499346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.499354 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.499398 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:07.499540 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.500204 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.501796 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.502173 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.502252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.502261 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.502399 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.502431 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 19:17:07.502492 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.499002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:41577 (local address 127.1.1.195:45821)
0620 19:17:07.499090 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.499099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.499137 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.499402 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.499403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.499404 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:07.499475 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.499476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.500274 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.500274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.501738 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.501740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.502256 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.502257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.502323 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.502326 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.502393 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.502427 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":44,"threads_started":1}
I20260620 19:17:07.502616 2513 leader_election.cc:302] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a92ef627f564481b2a94eb00eb3859d, f3f102d3d20147949c8f9997a41d50a8; no voters:
I20260620 19:17:07.502655 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae9d9bc8010458c87195c7a5a290cad" candidate_uuid: "f3f102d3d20147949c8f9997a41d50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192902ec9a894645ae0e1a7015f5027a" is_pre_election: true
I20260620 19:17:07.502732 2651 raft_consensus.cc:2781] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:07.502763 2651 raft_consensus.cc:482] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:07.502774 2441 raft_consensus.cc:2454] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f102d3d20147949c8f9997a41d50a8 in term 0.
I20260620 19:17:07.502789 2651 raft_consensus.cc:3037] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.503321 2651 raft_consensus.cc:504] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.503423 2651 leader_election.cc:288] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709), 192902ec9a894645ae0e1a7015f5027a (127.1.1.194:41577)
I20260620 19:17:07.503538 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae9d9bc8010458c87195c7a5a290cad" candidate_uuid: "f3f102d3d20147949c8f9997a41d50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192902ec9a894645ae0e1a7015f5027a"
I20260620 19:17:07.503539 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae9d9bc8010458c87195c7a5a290cad" candidate_uuid: "f3f102d3d20147949c8f9997a41d50a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92ef627f564481b2a94eb00eb3859d"
I20260620 19:17:07.503593 2441 raft_consensus.cc:3037] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.503593 2301 raft_consensus.cc:3037] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:07.504575 2301 raft_consensus.cc:2454] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f102d3d20147949c8f9997a41d50a8 in term 1.
I20260620 19:17:07.504575 2441 raft_consensus.cc:2454] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f102d3d20147949c8f9997a41d50a8 in term 1.
I20260620 19:17:07.504712 2515 leader_election.cc:302] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192902ec9a894645ae0e1a7015f5027a, f3f102d3d20147949c8f9997a41d50a8; no voters:
I20260620 19:17:07.504803 2651 raft_consensus.cc:2781] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:07.504928 2651 raft_consensus.cc:686] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [term 1 LEADER]: Becoming Leader. State: Replica: f3f102d3d20147949c8f9997a41d50a8, State: Running, Role: LEADER
I20260620 19:17:07.504988 2651 consensus_queue.cc:237] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } }
I20260620 19:17:07.505281 2686 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: TxnStatusTablet state changed. Reason: New leader f3f102d3d20147949c8f9997a41d50a8. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.505350 2686 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:07.505491 2688 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:17:07.505589 2688 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:17:07.505513 2144 catalog_manager.cc:5495] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } health_report { overall_health: HEALTHY } } }
I20260620 19:17:07.511580 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.509443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.1.1.195:36703)
0620 19:17:07.509482 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.509484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.509509 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:17:07.509578 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.509579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.509593 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.509650 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:07.509731 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.510131 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.510831 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.511027 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.511104 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.511279 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.511371 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.511446 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.511509 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.511526 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:17:07.511601 2681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.509393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36703 (local address 127.0.0.1:56640)
0620 19:17:07.509472 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.509482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.509525 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.509654 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.509654 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.509655 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:07.509686 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.509688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.510201 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.510202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.510754 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.510755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.511106 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.511107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.511167 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.511181 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.511433 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.511442 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.511505 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.511540 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 19:17:07.512391 2301 raft_consensus.cc:1261] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Refusing update from remote peer f3f102d3d20147949c8f9997a41d50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.512457 2441 raft_consensus.cc:1261] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Refusing update from remote peer f3f102d3d20147949c8f9997a41d50a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.512677 2651 consensus_queue.cc:1035] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.512809 2686 consensus_queue.cc:1035] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.513329 2657 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: TxnStatusTablet state changed. Reason: New leader f3f102d3d20147949c8f9997a41d50a8. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.513407 2657 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.513644 2654 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: TxnStatusTablet state changed. Reason: New leader f3f102d3d20147949c8f9997a41d50a8. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.513728 2654 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.514003 2651 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.514070 2651 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:07.514422 2654 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.514490 2654 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 8a92ef627f564481b2a94eb00eb3859d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.514446 2651 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.514525 2651 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P f3f102d3d20147949c8f9997a41d50a8: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:07.515033 2657 tablet_replica.cc:439] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3f102d3d20147949c8f9997a41d50a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 } } peers { permanent_uuid: "192902ec9a894645ae0e1a7015f5027a" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41577 } } peers { permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 } } }
I20260620 19:17:07.515110 2657 tablet_replica.cc:442] T 4ae9d9bc8010458c87195c7a5a290cad P 192902ec9a894645ae0e1a7015f5027a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:07.519167 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.517064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36703 (local address 127.1.1.194:44967)
0620 19:17:07.517191 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.517199 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.517252 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.517357 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.517357 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.517358 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:07.517385 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.517386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.518066 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.518066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.518530 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.518531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.518906 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.518907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.518937 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.518947 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.519069 (+ 122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.519070 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.519098 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.519125 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
I20260620 19:17:07.519168 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.517137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:44967 (local address 127.1.1.195:36703)
0620 19:17:07.517169 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.517170 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.517254 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:17:07.517297 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.517298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.517305 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.517356 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:07.517415 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.517999 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.518590 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.518843 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.518901 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.519000 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.519024 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.519065 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.519100 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.519113 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:07.521173 2301 raft_consensus.cc:1261] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Refusing update from remote peer 192902ec9a894645ae0e1a7015f5027a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.521193 2581 raft_consensus.cc:1261] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Refusing update from remote peer 192902ec9a894645ae0e1a7015f5027a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:07.521364 2657 consensus_queue.cc:1035] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a92ef627f564481b2a94eb00eb3859d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.521423 2655 consensus_queue.cc:1035] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f102d3d20147949c8f9997a41d50a8" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:07.528438 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.525803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:41577 (local address 127.1.1.195:56421)
0620 19:17:07.525948 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:07.525961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:07.526024 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:07.526152 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:07.526153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:07.526154 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:07.526213 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.526221 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.526836 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.526837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.527688 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:07.527690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.528091 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.528092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:07.528132 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.528142 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.528309 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.528310 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:17:07.528342 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 19:17:07.528369 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":73,"threads_started":1}
I20260620 19:17:07.528435 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.525977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:56421 (local address 127.1.1.194:41577)
0620 19:17:07.526015 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.526017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.526026 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:17:07.526084 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.526085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.526096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.526149 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:07.526287 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.526763 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.527747 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.528020 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.528080 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.528224 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:07.528254 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:07.528304 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.528345 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.528362 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:17:07.540102 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:07.540194 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:07.540788 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7ac7080b0c342eaa6ebe9f9490dabfd: Sending DeleteTablet for 3 replicas of tablet ed8a061d83224b44bd471ef202fac745
I20260620 19:17:07.541010 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a061d83224b44bd471ef202fac745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39032
I20260620 19:17:07.541127 2669 tablet_replica.cc:330] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: stopping tablet replica
I20260620 19:17:07.541188 2669 raft_consensus.cc:2229] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:07.541257 2669 raft_consensus.cc:2258] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:07.541024 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a061d83224b44bd471ef202fac745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:56628
I20260620 19:17:07.541450 2671 tablet_replica.cc:330] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: stopping tablet replica
I20260620 19:17:07.541502 2671 raft_consensus.cc:2229] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:07.541544 2671 raft_consensus.cc:2258] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:07.541555 2669 ts_tablet_manager.cc:1905] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.541841 2671 ts_tablet_manager.cc:1905] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.543080 2669 ts_tablet_manager.cc:1918] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:07.543124 2669 log.cc:1196] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/ed8a061d83224b44bd471ef202fac745
I20260620 19:17:07.543249 2671 ts_tablet_manager.cc:1918] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:07.543292 2671 log.cc:1196] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/ed8a061d83224b44bd471ef202fac745
I20260620 19:17:07.543329 2669 ts_tablet_manager.cc:1939] T ed8a061d83224b44bd471ef202fac745 P 192902ec9a894645ae0e1a7015f5027a: Deleting consensus metadata
I20260620 19:17:07.543538 2671 ts_tablet_manager.cc:1939] T ed8a061d83224b44bd471ef202fac745 P f3f102d3d20147949c8f9997a41d50a8: Deleting consensus metadata
I20260620 19:17:07.543633 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8a061d83224b44bd471ef202fac745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:07 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:48648
I20260620 19:17:07.543635 2131 catalog_manager.cc:4841] TS 192902ec9a894645ae0e1a7015f5027a (127.1.1.194:41577): tablet ed8a061d83224b44bd471ef202fac745 (table test-table [id=c53a3e2ae01e45288348e4e5ae450530]) successfully deleted
I20260620 19:17:07.543705 2670 tablet_replica.cc:330] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: stopping tablet replica
I20260620 19:17:07.543744 2670 raft_consensus.cc:2229] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:07.543784 2670 raft_consensus.cc:2258] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:07.543953 2129 catalog_manager.cc:4841] TS f3f102d3d20147949c8f9997a41d50a8 (127.1.1.195:36703): tablet ed8a061d83224b44bd471ef202fac745 (table test-table [id=c53a3e2ae01e45288348e4e5ae450530]) successfully deleted
I20260620 19:17:07.544111 2670 ts_tablet_manager.cc:1905] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:07.545529 2670 ts_tablet_manager.cc:1918] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:07.545585 2670 log.cc:1196] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/ed8a061d83224b44bd471ef202fac745
I20260620 19:17:07.545783 2670 ts_tablet_manager.cc:1939] T ed8a061d83224b44bd471ef202fac745 P 8a92ef627f564481b2a94eb00eb3859d: Deleting consensus metadata
I20260620 19:17:07.546038 2129 catalog_manager.cc:4841] TS 8a92ef627f564481b2a94eb00eb3859d (127.1.1.193:36709): tablet ed8a061d83224b44bd471ef202fac745 (table test-table [id=c53a3e2ae01e45288348e4e5ae450530]) successfully deleted
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:07.551838 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.543996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.1.1.193:36709)
0620 19:17:07.544030 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.544032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.544279 (+ 247us) server_negotiation.cc:407] Connection header received
0620 19:17:07.544344 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.544345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.544356 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.544405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.544910 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.545446 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.546271 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.546445 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.546504 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.549160 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.549167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.549167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.549174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.550737 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.551088 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.551092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.551092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.551159 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.551358 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.551359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.551359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.551497 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.551633 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.551689 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.551755 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:17:07.552009 2341 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48650
I20260620 19:17:07.558717 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:07.552791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60270 (local address 127.1.1.254:34605)
0620 19:17:07.552836 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:07.552838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:07.552885 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:17:07.552926 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:07.552927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:07.552936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:07.552995 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:07.553475 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.554198 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.554846 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:07.555149 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:07.555211 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:07.556216 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:07.556217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:07.556218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:07.556223 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:07.557683 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.558018 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.558019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.558019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.558076 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:07.558298 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:07.558299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:07.558299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:07.558412 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:07.558492 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:07.558566 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:17:07.558631 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 19:17:07.558876 2142 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60270
I20260620 19:17:07.559341 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2216
I20260620 19:17:07.564997 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2356
I20260620 19:17:07.571162 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2496
I20260620 19:17:07.576538 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2112
2026-06-20T19:17:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2200 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2709](info): set up 2 sockets
Jun 20 19:17:07 dist-test-slave-mwkb krb5kdc[2709](info): commencing operation
krb5kdc: starting...
W20260620 19:17:08.548210 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:08 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:08.649269 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38845
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:38845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:08.731732 2722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:08.731998 2722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:08.732038 2722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:08.733960 2722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:08.734019 2722 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:17:08.734042 2722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:08.734072 2722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:08.734109 2722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:08.734135 2722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:08.736780 2722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:38845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:08.737112 2722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:08.737357 2722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:08.742062 2727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:08.742264 2728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:08.742167 2730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:08.742242 2722 server_base.cc:1034] running on GCE node
I20260620 19:17:08.742576 2722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:08.742916 2722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:08.744084 2722 hybrid_clock.cc:648] HybridClock initialized: now 1781983028744059 us; error 41 us; skew 500 ppm
Jun 20 19:17:08 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:08.746433 2722 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:08.746850 2722 webserver.cc:460] Webserver started at http://127.1.1.254:44843/ using document root <none> and password file <none>
I20260620 19:17:08.747051 2722 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:08.747113 2722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:08.747231 2722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:08.748121 2722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "65aa39f8a7a646d697f0e9735051e6ee"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "8e1d65593676160dedd1ecd836d1cc77"
server_key_iv: "e5cf06639a0257e4969d87fa9f996dcf"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.748441 2722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "65aa39f8a7a646d697f0e9735051e6ee"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "8e1d65593676160dedd1ecd836d1cc77"
server_key_iv: "e5cf06639a0257e4969d87fa9f996dcf"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.749758 2722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:08.750504 2737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:08.750661 2722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:08.750757 2722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "65aa39f8a7a646d697f0e9735051e6ee"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "8e1d65593676160dedd1ecd836d1cc77"
server_key_iv: "e5cf06639a0257e4969d87fa9f996dcf"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.750833 2722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:08.765542 2722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:08.767020 2722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:08.767154 2722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:08.771288 2722 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:38845
I20260620 19:17:08.771330 2799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:38845 every 8 connection(s)
I20260620 19:17:08.771612 2722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:08.772090 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:08.772835 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2722
I20260620 19:17:08.772903 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:08.773036 1031 external_mini_cluster.cc:1428] Setting key a4374f731c5c3c27c7fbc6f21cfbe65d
I20260620 19:17:08.774281 2800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Bootstrap starting.
I20260620 19:17:08.774931 2800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:08.775187 2800 log.cc:824] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:08.775736 2800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: No bootstrap required, opened a new log
I20260620 19:17:08.777750 2800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } }
I20260620 19:17:08.777899 2800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:08.777935 2800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65aa39f8a7a646d697f0e9735051e6ee, State: Initialized, Role: FOLLOWER
I20260620 19:17:08.778057 2800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } }
I20260620 19:17:08.778136 2800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:08.778177 2800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:08.778218 2800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:08.778789 2800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } }
I20260620 19:17:08.778867 2800 leader_election.cc:302] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65aa39f8a7a646d697f0e9735051e6ee; no voters:
Jun 20 19:17:08 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:08.779035 2800 leader_election.cc:288] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:08.779060 2805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:08.779165 2805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [term 1 LEADER]: Becoming Leader. State: Replica: 65aa39f8a7a646d697f0e9735051e6ee, State: Running, Role: LEADER
I20260620 19:17:08.779258 2800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:08.779244 2805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } }
I20260620 19:17:08.779520 2805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65aa39f8a7a646d697f0e9735051e6ee. Latest consensus state: current_term: 1 leader_uuid: "65aa39f8a7a646d697f0e9735051e6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } } }
I20260620 19:17:08.779600 2805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:08.779532 2806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65aa39f8a7a646d697f0e9735051e6ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65aa39f8a7a646d697f0e9735051e6ee" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38845 } } }
I20260620 19:17:08.779786 2806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:08.779811 2810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:08.780263 2810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:08.783023 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:08.773468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.1.1.254:38845)
0620 19:17:08.773616 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:08.773623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:08.773654 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:17:08.774114 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:08.774120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:08.774171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:08.774271 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:08.774942 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.775757 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.776727 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.777062 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.777182 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:08.779481 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:08.779485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:08.779485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:08.779497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:08.781659 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.782002 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.782005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.782005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.782096 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.782295 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.782296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.782296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.782502 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:08.782628 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:08.782746 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:17:08.782879 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260620 19:17:08.785523 2810 catalog_manager.cc:1347] Generated new cluster ID: d0fee79e280a46819a718477d2295fef
I20260620 19:17:08.785578 2810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:08.793860 2810 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:08.794267 2810 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:08.798206 2810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Generated new TSK 0
I20260620 19:17:08.798352 2810 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:08.823722 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38845
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:08.898229 2826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:08.898409 2826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:08.898432 2826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:08.899400 2826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:08.899431 2826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:08.899451 2826 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:17:08.899461 2826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:08.899475 2826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:08.900830 2826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:38845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:08.900996 2826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:08.901213 2826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:08.905534 2832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:08.905503 2834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:08.905649 2826 server_base.cc:1034] running on GCE node
W20260620 19:17:08.905511 2831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:08.905870 2826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:08.906136 2826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:08.907274 2826 hybrid_clock.cc:648] HybridClock initialized: now 1781983028907259 us; error 27 us; skew 500 ppm
Jun 20 19:17:08 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:08.909439 2826 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:08.909826 2826 webserver.cc:460] Webserver started at http://127.1.1.193:39621/ using document root <none> and password file <none>
I20260620 19:17:08.910027 2826 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:08.910065 2826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:08.910195 2826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:08.911135 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "16580efd9fe57c961ba68d25d1a4b58e"
server_key_iv: "720630925ccada23468850c5d1ece60c"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.911459 2826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "16580efd9fe57c961ba68d25d1a4b58e"
server_key_iv: "720630925ccada23468850c5d1ece60c"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.912607 2826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:08.913256 2841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:08.913409 2826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:08.913486 2826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
format_stamp: "Formatted at 2026-06-20 19:17:08 on dist-test-slave-mwkb"
server_key: "16580efd9fe57c961ba68d25d1a4b58e"
server_key_iv: "720630925ccada23468850c5d1ece60c"
server_key_version: "encryptionkey@0"
I20260620 19:17:08.913547 2826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:08.920253 2826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:08.922145 2826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:08.922264 2826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:08.922768 2826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:08.922807 2826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:08.922827 2826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:08.922857 2826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:08.929177 2826 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:35079
Jun 20 19:17:08 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:08.929490 2826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:08.929643 2956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:35079 every 8 connection(s)
I20260620 19:17:08.932338 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:08.923219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:53799 (local address 127.1.1.254:38845)
0620 19:17:08.923281 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:08.923283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:08.923446 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:17:08.923871 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:08.923873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:08.923884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:08.923955 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:08.924804 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.925558 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.927059 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.927373 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.927438 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:08.929982 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:08.929984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:08.929984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:08.929991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:08.931064 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.931456 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.931457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.931457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.931513 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.931775 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.931776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.931776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.931877 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:08.931944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:08.932184 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:17:08.932259 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:17:08.932511 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:08.923194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.193:53799)
0620 19:17:08.923349 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:08.923383 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:08.923777 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:08.924064 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:08.924069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:08.924472 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:08.924717 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:08.924723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.925660 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.925662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:08.926976 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:08.926979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.927453 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.927454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:08.927593 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:08.928193 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:08.928203 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:08.929896 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:08.931159 (+ 1263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:08.931166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:08.931168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:08.931374 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:08.931617 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:08.931618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:08.931618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:08.931714 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:08.931935 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:08.931941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:08.932064 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 19:17:08.932163 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:17:08.932294 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260620 19:17:08.932934 2957 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38845
I20260620 19:17:08.933007 2957 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:08.933197 2957 heartbeater.cc:511] Master 127.1.1.254:38845 requested a full tablet report, sending...
I20260620 19:17:08.933705 2753 ts_manager.cc:194] Registered new tserver with Master: e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079)
I20260620 19:17:08.934278 2753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:53799
I20260620 19:17:08.937376 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2826
I20260620 19:17:08.937469 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:08.937624 1031 external_mini_cluster.cc:1428] Setting key 3c7224d7b5cf56bc318ca70ffb8e9fa4
I20260620 19:17:08.938827 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:08.933869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:57325 (local address 127.1.1.254:38845)
0620 19:17:08.934409 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:08.934411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:08.934417 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:08.934429 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:08.934429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:08.934436 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:08.934495 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:08.934775 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.935159 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.935770 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.935948 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.936008 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:08.936690 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:08.936691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:08.936691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:08.936697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:08.937884 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.938166 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.938167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.938167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.938233 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:08.938413 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:08.938414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:08.938414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:08.938486 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:08.938523 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:08.938683 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:17:08.938756 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529}
I20260620 19:17:08.938884 2963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:08.933820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.193:57325)
0620 19:17:08.933942 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:08.933954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:08.933999 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:08.934530 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:08.934531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:08.934667 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:08.934710 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:08.934711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.935235 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.935236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:08.935700 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:08.935701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:08.936018 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:08.936019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:08.936092 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:08.936387 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:08.936390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:08.936614 (+ 224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:08.937938 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:08.937938 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:08.937939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:08.938105 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:08.938299 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:08.938300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:08.938300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:08.938359 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:08.938539 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:08.938540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:08.938585 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:17:08.938654 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:17:08.938750 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:08.957731 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38845
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:09.030230 2966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:09.030450 2966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:09.030490 2966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:09.031605 2966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:09.031667 2966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:09.031708 2966 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:17:09.031747 2966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:09.031783 2966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:09.033205 2966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:38845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:09.033430 2966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:09.033658 2966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:09.038106 2974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:09.038189 2972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:09.038106 2971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:09.038287 2966 server_base.cc:1034] running on GCE node
I20260620 19:17:09.038496 2966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:09.038731 2966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:09.039891 2966 hybrid_clock.cc:648] HybridClock initialized: now 1781983029039881 us; error 35 us; skew 500 ppm
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:09.042136 2966 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:09.042482 2966 webserver.cc:460] Webserver started at http://127.1.1.194:39939/ using document root <none> and password file <none>
I20260620 19:17:09.042685 2966 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:09.042740 2966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:09.042869 2966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:09.043730 2966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "128adcd646e9442f80bb3c831fdb380d"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "1141c0ff20e464f3d2db2df3a8aaf1c2"
server_key_iv: "bcf5e01ba3260b05b2a61e978ee48aae"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.044039 2966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "128adcd646e9442f80bb3c831fdb380d"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "1141c0ff20e464f3d2db2df3a8aaf1c2"
server_key_iv: "bcf5e01ba3260b05b2a61e978ee48aae"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.045117 2966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:09.045691 2981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.045841 2966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:09.045918 2966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "128adcd646e9442f80bb3c831fdb380d"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "1141c0ff20e464f3d2db2df3a8aaf1c2"
server_key_iv: "bcf5e01ba3260b05b2a61e978ee48aae"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.046005 2966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:09.080991 2966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:09.082794 2966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:09.082926 2966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:09.083305 2966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:09.083360 2966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.083397 2966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:09.083431 2966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:09.089488 2966 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:37279
I20260620 19:17:09.089556 3096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:37279 every 8 connection(s)
I20260620 19:17:09.089835 2966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:09.091348 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 2966
I20260620 19:17:09.091447 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:09.091607 1031 external_mini_cluster.cc:1428] Setting key 3b6bead50ace4ed9f8f107d98280dbe8
I20260620 19:17:09.092046 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.083738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:36121 (local address 127.1.1.254:38845)
0620 19:17:09.083780 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.083783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.083946 (+ 163us) server_negotiation.cc:407] Connection header received
0620 19:17:09.084328 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.084330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.084343 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.084412 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.085088 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.085840 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.086713 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.087004 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.087075 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.089310 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.089311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.089312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.089318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.090694 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.091067 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.091068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.091069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.091130 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.091375 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.091376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.091376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.091513 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.091573 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.091850 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.091949 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:17:09.092175 2990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.083724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.194:36121)
0620 19:17:09.083856 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.083883 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.084232 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.084457 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.084459 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.084731 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:09.084971 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.084976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.085928 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.085929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.086645 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.086647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.087089 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.087090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.087207 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.087691 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:09.087707 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:09.089228 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:09.090785 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.090792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.090793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.090995 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.091197 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.091198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.091198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.091302 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.091565 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:09.091570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:09.091727 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.091829 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.091973 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260620 19:17:09.092558 3097 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38845
I20260620 19:17:09.092625 3097 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:09.092792 3097 heartbeater.cc:511] Master 127.1.1.254:38845 requested a full tablet report, sending...
I20260620 19:17:09.093122 2753 ts_manager.cc:194] Registered new tserver with Master: 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:09.093565 2753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:36121
I20260620 19:17:09.098994 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.093440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.194:39025)
0620 19:17:09.093561 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.093569 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.093651 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.093787 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.093787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.094239 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:09.094281 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.094283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.094933 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.094933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.095370 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.095371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.095826 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.095827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.095877 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.096288 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:09.096292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:09.096543 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:09.098034 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.098035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.098035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.098196 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.098443 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.098443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.098444 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.098487 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.098798 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:09.098799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:09.098827 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.098872 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.098934 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260620 19:17:09.099053 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.093480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:39025 (local address 127.1.1.254:38845)
0620 19:17:09.093515 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.093516 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.093631 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:17:09.093709 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.093710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.093719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.093771 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.094349 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.094865 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.095436 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.095740 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.095807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.096619 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.096620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.096621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.096627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.097956 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.098308 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.098309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.098309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.098370 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.098536 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.098536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.098537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.098735 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.098808 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.098897 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.098964 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:09.115206 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38845
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:09.187007 3106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:09.187171 3106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:09.187192 3106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:09.188200 3106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:09.188235 3106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:09.188252 3106 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:17:09.188293 3106 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:17:09.188313 3106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:09.189608 3106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:38845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:09.189797 3106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:09.189972 3106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:09.194530 3111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:09.194548 3114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:09.194685 3112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:09.194837 3106 server_base.cc:1034] running on GCE node
I20260620 19:17:09.194957 3106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:09.195113 3106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:09.196209 3106 hybrid_clock.cc:648] HybridClock initialized: now 1781983029196197 us; error 25 us; skew 500 ppm
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:09.198452 3106 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:09.198827 3106 webserver.cc:460] Webserver started at http://127.1.1.195:46817/ using document root <none> and password file <none>
I20260620 19:17:09.199000 3106 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:09.199041 3106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:09.199160 3106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:09.200076 3106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "fc8801cf0b734771a6f3c436da852aa7"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "b7fad7a905f83e30e8c102440092499c"
server_key_iv: "4fe6026f82c9b0d88dbb7f748ac0d29f"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.200407 3106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "fc8801cf0b734771a6f3c436da852aa7"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "b7fad7a905f83e30e8c102440092499c"
server_key_iv: "4fe6026f82c9b0d88dbb7f748ac0d29f"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.201841 3106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:09.202564 3121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.202716 3106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:09.202800 3106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "fc8801cf0b734771a6f3c436da852aa7"
format_stamp: "Formatted at 2026-06-20 19:17:09 on dist-test-slave-mwkb"
server_key: "b7fad7a905f83e30e8c102440092499c"
server_key_iv: "4fe6026f82c9b0d88dbb7f748ac0d29f"
server_key_version: "encryptionkey@0"
I20260620 19:17:09.202873 3106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:09.213697 3106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:09.215260 3106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:09.215408 3106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:09.215910 3106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:09.215965 3106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.216004 3106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:09.216030 3106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:09.222378 3106 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:45043
I20260620 19:17:09.222441 3236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:45043 every 8 connection(s)
I20260620 19:17:09.222697 3106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:09.224874 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.216272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:59269 (local address 127.1.1.254:38845)
0620 19:17:09.216313 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.216315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.216666 (+ 351us) server_negotiation.cc:407] Connection header received
0620 19:17:09.217153 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.217154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.217165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.217227 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.217903 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.218630 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.220071 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.220423 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.220505 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.222505 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.222507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.222507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.222514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.223909 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.224226 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.224227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.224227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.224275 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.224447 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.224448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.224448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.224531 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.224575 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.224767 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.224818 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:09.225020 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.216280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.195:59269)
0620 19:17:09.216433 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.216462 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.217042 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.217286 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.217289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.217557 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:09.217809 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.217814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.218718 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.218719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.219976 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.219978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.220510 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.220511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.220641 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.221103 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:09.221112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:09.222401 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:09.223983 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.223988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.223989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.224167 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.224323 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.224323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.224323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.224402 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.224576 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:09.224579 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:09.224666 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.224743 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.224836 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 19:17:09.225497 3237 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38845
I20260620 19:17:09.225574 3237 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:09.225742 3237 heartbeater.cc:511] Master 127.1.1.254:38845 requested a full tablet report, sending...
I20260620 19:17:09.226030 2754 ts_manager.cc:194] Registered new tserver with Master: fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043)
I20260620 19:17:09.226336 2754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:59269
I20260620 19:17:09.229190 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3106
I20260620 19:17:09.229288 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:09.229457 1031 external_mini_cluster.cc:1428] Setting key 9dd0fd832fd2141ac2eb286e2ab863b6
I20260620 19:17:09.230278 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:09.231303 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.226365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.1.1.195:46933)
0620 19:17:09.226468 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.226478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.226537 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.226657 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.226658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.226822 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:09.226867 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.226869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.227426 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.227426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.227868 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.227869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.228136 (+ 267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.228137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.228190 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.228492 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:09.228495 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:09.228712 (+ 217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:09.230389 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.230390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.230390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.230580 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.230786 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.230787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.230787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.230836 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.231088 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:09.231089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:09.231119 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.231173 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.231240 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":63,"threads_started":1}
I20260620 19:17:09.231416 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.226426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:46933 (local address 127.1.1.254:38845)
0620 19:17:09.226457 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.226459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.226519 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:17:09.226588 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.226589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.226598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.226651 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.226931 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.227350 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.227927 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.228083 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.228128 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.228774 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.228775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.228775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.228780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.230319 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.230645 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.230646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.230646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.230708 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.230892 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.230893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.230893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.231024 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.231093 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.231203 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.231279 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:09.251483 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.245363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.1.1.254:38845)
0620 19:17:09.245409 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.245411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.245431 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:17:09.245481 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.245481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.245488 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.245532 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.246008 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.246469 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.247110 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.247278 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.247323 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.249380 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.249382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.249382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.249393 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.250451 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.250783 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.250784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.250785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.250836 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.251020 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.251021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.251021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.251116 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.251220 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.251316 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.251388 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:17:09.252785 2754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:09.254281 2754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:09.260330 3252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.255926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.0.0.1:38214)
0620 19:17:09.256060 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.256090 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.256147 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.256418 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.256421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.256423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.256471 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.256472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.257894 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.257895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.259256 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.259258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.260036 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.260112 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260135 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.260202 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.260261 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":35,"thread_start_us":52,"threads_started":1}
I20260620 19:17:09.260357 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.256022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.1.1.193:35079)
0620 19:17:09.256180 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.256185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.256197 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:17:09.256237 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.256238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.256253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.256335 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.256556 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.257807 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.259349 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.259952 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260024 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260075 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.260228 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.260265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260620 19:17:09.260568 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.256012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.1.1.194:37279)
0620 19:17:09.256205 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.256209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.256293 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:17:09.256424 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.256426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.256441 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.256493 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.256693 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.257800 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.259333 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.259976 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260056 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260111 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.260289 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.260326 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260620 19:17:09.260829 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.255926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37279 (local address 127.0.0.1:46700)
0620 19:17:09.256237 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.256245 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.256319 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.256579 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.256580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.256582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.256628 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.256629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.257895 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.257895 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.259255 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.259258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.260141 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.260203 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260206 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.260722 (+ 516us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.260770 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":74,"spinlock_wait_cycles":384,"thread_start_us":48,"threads_started":1}
I20260620 19:17:09.261130 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.256251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.1.1.195:45043)
0620 19:17:09.256381 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.256385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.256392 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:09.256439 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.256440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.256453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.256527 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.258014 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.258656 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.259753 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.260458 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260524 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260572 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.260995 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.261031 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 19:17:09.260972 2891 tablet_service.cc:1467] Processing CreateTablet for tablet 42926cd5237544fe9de63424894f8ebc (DEFAULT_TABLE table=test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.260846 3255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.256090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45043 (local address 127.0.0.1:44388)
0620 19:17:09.256262 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.256271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.256315 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.257901 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.257902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.257905 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.257945 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.257946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.258733 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.258734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.259661 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.259664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.260605 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.260606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.260670 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.260673 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.260722 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.260761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260620 19:17:09.261350 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42926cd5237544fe9de63424894f8ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.263727 3257 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Bootstrap starting.
I20260620 19:17:09.264031 3031 tablet_service.cc:1467] Processing CreateTablet for tablet 42926cd5237544fe9de63424894f8ebc (DEFAULT_TABLE table=test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.264329 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42926cd5237544fe9de63424894f8ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.264341 3257 tablet_bootstrap.cc:654] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.264597 3257 log.cc:824] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:09.265311 3257 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: No bootstrap required, opened a new log
I20260620 19:17:09.265391 3257 ts_tablet_manager.cc:1397] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:09.265583 3171 tablet_service.cc:1467] Processing CreateTablet for tablet 42926cd5237544fe9de63424894f8ebc (DEFAULT_TABLE table=test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.265861 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42926cd5237544fe9de63424894f8ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.266685 3257 raft_consensus.cc:348] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.266815 3257 raft_consensus.cc:374] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.266855 3257 raft_consensus.cc:729] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.266953 3257 consensus_queue.cc:260] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.267215 3257 ts_tablet_manager.cc:1428] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:17:09.267220 2957 heartbeater.cc:503] Master 127.1.1.254:38845 was elected leader, sending a full tablet report...
I20260620 19:17:09.267545 3260 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Bootstrap starting.
I20260620 19:17:09.268010 3260 tablet_bootstrap.cc:654] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.268261 3260 log.cc:824] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:09.268559 3262 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Bootstrap starting.
I20260620 19:17:09.269019 3260 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: No bootstrap required, opened a new log
I20260620 19:17:09.269088 3260 ts_tablet_manager.cc:1397] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:09.269364 3262 tablet_bootstrap.cc:654] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.269615 3262 log.cc:824] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:09.270234 3262 tablet_bootstrap.cc:492] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: No bootstrap required, opened a new log
I20260620 19:17:09.270304 3262 ts_tablet_manager.cc:1397] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:17:09.270511 3260 raft_consensus.cc:348] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.270624 3260 raft_consensus.cc:374] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.270645 3260 raft_consensus.cc:729] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128adcd646e9442f80bb3c831fdb380d, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.270746 3260 consensus_queue.cc:260] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.270979 3260 ts_tablet_manager.cc:1428] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:09.271013 3097 heartbeater.cc:503] Master 127.1.1.254:38845 was elected leader, sending a full tablet report...
I20260620 19:17:09.272089 3262 raft_consensus.cc:348] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.272213 3262 raft_consensus.cc:374] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.272251 3262 raft_consensus.cc:729] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.272341 3262 consensus_queue.cc:260] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.272536 3262 ts_tablet_manager.cc:1428] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:17:09.272578 3237 heartbeater.cc:503] Master 127.1.1.254:38845 was elected leader, sending a full tablet report...
W20260620 19:17:09.340404 3098 tablet.cc:2369] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:09.430094 2958 tablet.cc:2369] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:09.473531 3238 tablet.cc:2369] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:17:09.627596 3265 raft_consensus.cc:482] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:09.627696 3265 raft_consensus.cc:504] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.628005 3265 leader_election.cc:288] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:09.630676 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.628231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:52331 (local address 127.1.1.194:37279)
0620 19:17:09.628364 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.628366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.628377 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:09.628443 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.628444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.628455 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.628500 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.628654 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.629062 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.629989 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.630320 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.630403 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.630413 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.630564 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.630597 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 19:17:09.630702 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.628217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:34365 (local address 127.1.1.193:35079)
0620 19:17:09.628259 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.628261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.628270 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:17:09.628425 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.628427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.628437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.628490 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.628636 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.629061 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.629999 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.630346 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.630426 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.630453 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.630596 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.630634 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:09.630698 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.628194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37279 (local address 127.1.1.195:52331)
0620 19:17:09.628331 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.628340 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.628390 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.628514 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.628514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.628516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.628584 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.628585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.629189 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.629190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.629905 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.629907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.630398 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.630399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.630485 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.630489 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.630547 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.630586 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260620 19:17:09.630941 3130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.628185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.1.1.195:34365)
0620 19:17:09.628232 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.628242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.628283 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.628507 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.628507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.628509 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.628573 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.628574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.629208 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.629209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.629912 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.629914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.630425 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.630426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.630527 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.630529 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.630591 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.630630 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":4}
I20260620 19:17:09.630975 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42926cd5237544fe9de63424894f8ebc" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d" is_pre_election: true
I20260620 19:17:09.631155 3051 raft_consensus.cc:2454] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:09.631232 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42926cd5237544fe9de63424894f8ebc" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" is_pre_election: true
I20260620 19:17:09.631353 2911 raft_consensus.cc:2454] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:09.631539 3124 leader_election.cc:302] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128adcd646e9442f80bb3c831fdb380d, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:09.631698 3265 raft_consensus.cc:2781] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:09.631755 3265 raft_consensus.cc:482] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:09.631772 3265 raft_consensus.cc:3037] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.632350 3265 raft_consensus.cc:504] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.632467 3265 leader_election.cc:288] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Requested vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:09.632582 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42926cd5237544fe9de63424894f8ebc" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
I20260620 19:17:09.632617 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42926cd5237544fe9de63424894f8ebc" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d"
I20260620 19:17:09.632679 3051 raft_consensus.cc:3037] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.632659 2911 raft_consensus.cc:3037] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.633443 3051 raft_consensus.cc:2454] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:09.633443 2911 raft_consensus.cc:2454] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:09.633584 3124 leader_election.cc:302] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128adcd646e9442f80bb3c831fdb380d, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:09.633713 3265 raft_consensus.cc:2781] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:09.633848 3265 raft_consensus.cc:686] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Becoming Leader. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Running, Role: LEADER
I20260620 19:17:09.633949 3265 consensus_queue.cc:237] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.634720 2753 catalog_manager.cc:5495] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "fc8801cf0b734771a6f3c436da852aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } health_report { overall_health: HEALTHY } } }
Jun 20 19:17:09 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983028, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:09.719864 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.709531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38845 (local address 127.0.0.1:46808)
0620 19:17:09.709730 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.709737 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.709789 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.709883 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.709884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.710161 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:09.710215 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.710217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.710827 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.710828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.711382 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.711384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.711696 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.711697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.711732 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.716605 (+ 4873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:09.716609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:09.717845 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:09.719001 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.719006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.719008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.719186 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.719332 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:09.719333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:09.719333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:09.719380 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:09.719591 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:09.719596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:09.719629 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.719703 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.719781 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":131,"threads_started":1}
I20260620 19:17:09.719918 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.709603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.1.1.254:38845)
0620 19:17:09.709684 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.709685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.709793 (+ 108us) server_negotiation.cc:407] Connection header received
0620 19:17:09.709819 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.709820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.709829 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.709873 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:09.710278 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.710766 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.711447 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.711632 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.711688 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.717921 (+ 6233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:09.717922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:09.717923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:09.717928 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:09.718943 (+ 1015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.719241 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.719241 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.719242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.719285 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:09.719435 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:09.719435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:09.719436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:09.719532 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:09.719597 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.719728 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.719824 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":62}
I20260620 19:17:09.721786 2753 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46808:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:17:09.721946 2753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:09.723119 2891 tablet_service.cc:1467] Processing CreateTablet for tablet b88ec0b41eef464c960f63e01cbb1e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=268bbd0507d84de48cbb25dee65aed00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:09.723124 3031 tablet_service.cc:1467] Processing CreateTablet for tablet b88ec0b41eef464c960f63e01cbb1e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=268bbd0507d84de48cbb25dee65aed00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:09.723270 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ec0b41eef464c960f63e01cbb1e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.723270 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ec0b41eef464c960f63e01cbb1e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.724625 3260 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: Bootstrap starting.
I20260620 19:17:09.724622 3257 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: Bootstrap starting.
I20260620 19:17:09.724707 3171 tablet_service.cc:1467] Processing CreateTablet for tablet b88ec0b41eef464c960f63e01cbb1e78 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=268bbd0507d84de48cbb25dee65aed00]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:17:09.724857 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b88ec0b41eef464c960f63e01cbb1e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.725041 3257 tablet_bootstrap.cc:654] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.725041 3260 tablet_bootstrap.cc:654] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.725663 3260 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: No bootstrap required, opened a new log
I20260620 19:17:09.725728 3260 ts_tablet_manager.cc:1397] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:09.725850 3257 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: No bootstrap required, opened a new log
I20260620 19:17:09.725894 3257 ts_tablet_manager.cc:1397] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:09.725888 3260 raft_consensus.cc:348] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.725963 3260 raft_consensus.cc:374] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.725991 3260 raft_consensus.cc:729] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128adcd646e9442f80bb3c831fdb380d, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.726032 3260 consensus_queue.cc:260] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.726043 3257 raft_consensus.cc:348] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.726130 3257 raft_consensus.cc:374] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.726135 3260 ts_tablet_manager.cc:1428] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.726161 3257 raft_consensus.cc:729] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.726207 3257 consensus_queue.cc:260] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.726302 3257 ts_tablet_manager.cc:1428] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.726315 3264 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.726342 3259 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.726403 3264 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.726414 3259 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.726483 3262 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: Bootstrap starting.
I20260620 19:17:09.726980 3262 tablet_bootstrap.cc:654] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.727517 3262 tablet_bootstrap.cc:492] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: No bootstrap required, opened a new log
I20260620 19:17:09.727555 3262 ts_tablet_manager.cc:1397] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:09.727650 3262 raft_consensus.cc:348] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.727702 3262 raft_consensus.cc:374] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.727715 3262 raft_consensus.cc:729] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.727762 3262 consensus_queue.cc:260] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.727846 3262 ts_tablet_manager.cc:1428] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.727893 3265 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.727981 3265 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.864487 3259 raft_consensus.cc:482] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:09.864578 3259 raft_consensus.cc:504] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.864885 3259 leader_election.cc:288] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279), fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043)
I20260620 19:17:09.868574 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.865044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37279 (local address 127.1.1.193:60179)
0620 19:17:09.865181 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.865192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.865256 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.865388 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.865388 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.865390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.865489 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.865491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.866324 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.866324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.867683 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.867686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.868353 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.868353 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.868408 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.868411 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.868450 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.868478 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":61,"thread_start_us":68,"threads_started":1}
I20260620 19:17:09.868584 3253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.865130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60179 (local address 127.1.1.194:37279)
0620 19:17:09.865177 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.865179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.865266 (+ 87us) server_negotiation.cc:407] Connection header received
0620 19:17:09.865299 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.865300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.865309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.865368 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.865554 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.866246 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.867759 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.868262 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.868333 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.868343 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.868475 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.868515 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:17:09.868794 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.865044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45043 (local address 127.1.1.193:53293)
0620 19:17:09.865200 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.865211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.865257 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.865387 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.865388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.865391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:09.865489 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.865491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.866324 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.866324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.867824 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.867826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.868561 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.868562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.868624 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.868628 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.868677 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.868715 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":61,"thread_start_us":67,"threads_started":1}
I20260620 19:17:09.868781 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ec0b41eef464c960f63e01cbb1e78" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d" is_pre_election: true
I20260620 19:17:09.868844 3051 raft_consensus.cc:2454] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 0.
I20260620 19:17:09.868870 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.865106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:53293 (local address 127.1.1.195:45043)
0620 19:17:09.865199 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.865201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.865265 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:17:09.865299 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.865299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.865309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.865372 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:09.865554 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.866258 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.867890 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.868486 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.868548 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.868558 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.868746 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.868788 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":41,"threads_started":1}
I20260620 19:17:09.868993 2844 leader_election.cc:302] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128adcd646e9442f80bb3c831fdb380d, e0d0c20cfea4447e8dfb6d7ec303482a; no voters:
I20260620 19:17:09.869212 3259 raft_consensus.cc:2781] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:09.869184 3191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ec0b41eef464c960f63e01cbb1e78" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8801cf0b734771a6f3c436da852aa7" is_pre_election: true
I20260620 19:17:09.869256 3259 raft_consensus.cc:482] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:09.869270 3259 raft_consensus.cc:3037] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.869284 3191 raft_consensus.cc:2454] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 0.
I20260620 19:17:09.869899 3259 raft_consensus.cc:504] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.870066 3259 leader_election.cc:288] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 election: Requested vote from peers 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279), fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043)
I20260620 19:17:09.870196 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ec0b41eef464c960f63e01cbb1e78" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d"
I20260620 19:17:09.870282 3051 raft_consensus.cc:3037] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.870306 3191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b88ec0b41eef464c960f63e01cbb1e78" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8801cf0b734771a6f3c436da852aa7"
I20260620 19:17:09.870393 3191 raft_consensus.cc:3037] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:09.871104 3051 raft_consensus.cc:2454] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 1.
I20260620 19:17:09.871161 3191 raft_consensus.cc:2454] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 1.
I20260620 19:17:09.871321 2845 leader_election.cc:302] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d0c20cfea4447e8dfb6d7ec303482a, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:09.871423 3259 raft_consensus.cc:2781] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:09.871538 3259 raft_consensus.cc:686] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 LEADER]: Becoming Leader. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Running, Role: LEADER
I20260620 19:17:09.871637 3259 consensus_queue.cc:237] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.872076 3281 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: TxnStatusTablet state changed. Reason: New leader e0d0c20cfea4447e8dfb6d7ec303482a. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.872164 3281 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:09.872376 2754 catalog_manager.cc:5495] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a reported cstate change: term changed from 0 to 1, leader changed from <none> to e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193). New cstate: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:09.872877 3283 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:17:09.873023 3283 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:17:09.907902 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.905095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.0.0.1:38220)
0620 19:17:09.905171 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.905184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.905235 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.905485 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.905486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.905488 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:09.905537 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.905544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.906447 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.906448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.906999 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.907001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.907459 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.907460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.907511 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.907525 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.907780 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.907785 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.907813 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.907848 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:17:09.907927 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.905129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.1.1.193:35079)
0620 19:17:09.905325 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.905327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.905334 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:09.905363 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.905363 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.905380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.905438 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:09.905604 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.906383 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.907080 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.907384 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.907446 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.907602 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.907718 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.907795 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.907817 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.907838 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260620 19:17:09.909619 3191 raft_consensus.cc:1261] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Refusing update from remote peer e0d0c20cfea4447e8dfb6d7ec303482a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:09.909631 3051 raft_consensus.cc:1261] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Refusing update from remote peer e0d0c20cfea4447e8dfb6d7ec303482a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:09.909919 3281 consensus_queue.cc:1035] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Connected to new peer: Peer: permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:09.910046 3259 consensus_queue.cc:1035] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:09.911092 3265 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: TxnStatusTablet state changed. Reason: New leader e0d0c20cfea4447e8dfb6d7ec303482a. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.911196 3265 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.911629 3264 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: TxnStatusTablet state changed. Reason: New leader e0d0c20cfea4447e8dfb6d7ec303482a. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.911718 3264 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.912724 3259 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.912817 3259 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:09.912874 3286 mvcc.cc:205] Tried to move back new op lower bound from 7299002490507300864 to 7299002490354626560. Current Snapshot: MvccSnapshot[applied={T|T < 7299002490507300864}]
I20260620 19:17:09.912933 3265 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.913013 3265 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P fc8801cf0b734771a6f3c436da852aa7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.913022 3281 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.913110 3281 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P e0d0c20cfea4447e8dfb6d7ec303482a: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:17:09.913232 3288 mvcc.cc:205] Tried to move back new op lower bound from 7299002490507300864 to 7299002490354626560. Current Snapshot: MvccSnapshot[applied={T|T < 7299002490507300864}]
I20260620 19:17:09.913497 3264 tablet_replica.cc:439] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } } }
I20260620 19:17:09.913576 3264 tablet_replica.cc:442] T b88ec0b41eef464c960f63e01cbb1e78 P 128adcd646e9442f80bb3c831fdb380d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:17:09.914036 3287 mvcc.cc:205] Tried to move back new op lower bound from 7299002490507300864 to 7299002490354626560. Current Snapshot: MvccSnapshot[applied={T|T < 7299002490507300864 or (T in {7299002490507300864})}]
I20260620 19:17:09.921218 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.918657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.1.1.195:45043)
0620 19:17:09.918702 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.918704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.918779 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:17:09.918844 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.918845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.918858 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.918907 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:09.919027 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.919449 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.920536 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.920704 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.920756 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.920908 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.920983 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.921035 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.921122 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.921142 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:17:09.926117 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.923497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:39813 (local address 127.1.1.193:35079)
0620 19:17:09.923535 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.923537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.923650 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:17:09.923677 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.923677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.923686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.923746 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:09.923840 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.924528 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.925353 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.925661 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.925728 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.925870 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.925900 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.925952 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.926030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.926046 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:17:09.926128 3298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.923429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.1.1.195:39813)
0620 19:17:09.923579 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.923593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.923643 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.923749 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.923749 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.923750 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:09.923799 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.923800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.924621 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.924622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.925252 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.925254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.925732 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.925733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.925779 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.925790 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.925982 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.925985 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.926027 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.926062 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260620 19:17:09.928390 3051 raft_consensus.cc:1261] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:09.928411 2911 raft_consensus.cc:1261] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:09.928632 3267 consensus_queue.cc:1035] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:09.928741 3265 consensus_queue.cc:1035] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:09.938565 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.936042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45043 (local address 127.1.1.193:33797)
0620 19:17:09.936186 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:09.936196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:09.936251 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:09.936361 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:09.936362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:09.936362 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:09.936390 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.936391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.937146 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.937147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.937843 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:09.937845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.938250 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.938250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:09.938285 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.938296 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.938460 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.938460 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:17:09.938491 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 19:17:09.938520 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260620 19:17:09.938573 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:09.936111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:33797 (local address 127.1.1.195:45043)
0620 19:17:09.936145 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:09.936147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:09.936242 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:17:09.936290 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:09.936291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:09.936299 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:09.936359 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:09.936424 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.937029 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.937931 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:09.938176 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:09.938207 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:09.938368 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:09.938399 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:09.938461 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:09.938496 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:17:09.938515 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:17:09.947948 2752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:09.948043 2752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:09.948683 2752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Sending DeleteTablet for 3 replicas of tablet 42926cd5237544fe9de63424894f8ebc
I20260620 19:17:09.949025 3031 tablet_service.cc:1514] Processing DeleteTablet for tablet 42926cd5237544fe9de63424894f8ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:09 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46700
I20260620 19:17:09.949162 2891 tablet_service.cc:1514] Processing DeleteTablet for tablet 42926cd5237544fe9de63424894f8ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:09 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260620 19:17:09.949288 3308 tablet_replica.cc:330] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: stopping tablet replica
I20260620 19:17:09.949371 3308 raft_consensus.cc:2229] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:09.949378 3171 tablet_service.cc:1514] Processing DeleteTablet for tablet 42926cd5237544fe9de63424894f8ebc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:09 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44388
I20260620 19:17:09.949416 3308 raft_consensus.cc:2258] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:09.949810 2752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:09.949941 2752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:09.951007 3031 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1e2b139064daca61ce5a61a18638a (DEFAULT_TABLE table=test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.951054 3171 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1e2b139064daca61ce5a61a18638a (DEFAULT_TABLE table=test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.951133 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1e2b139064daca61ce5a61a18638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.951174 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1e2b139064daca61ce5a61a18638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.952456 3262 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Bootstrap starting.
I20260620 19:17:09.952491 2891 tablet_service.cc:1467] Processing CreateTablet for tablet 5ad1e2b139064daca61ce5a61a18638a (DEFAULT_TABLE table=test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:09.952587 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ad1e2b139064daca61ce5a61a18638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:09.952759 3309 tablet_replica.cc:330] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: stopping tablet replica
I20260620 19:17:09.952827 3309 raft_consensus.cc:2229] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:09.952831 3262 tablet_bootstrap.cc:654] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.952862 3309 raft_consensus.cc:2258] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:09.953132 3260 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Bootstrap starting.
I20260620 19:17:09.953466 3260 tablet_bootstrap.cc:654] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.953845 3257 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Bootstrap starting.
I20260620 19:17:09.954066 3310 tablet_replica.cc:330] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: stopping tablet replica
I20260620 19:17:09.954149 3310 raft_consensus.cc:2229] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:09.954188 3257 tablet_bootstrap.cc:654] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:09.954228 3310 raft_consensus.cc:2258] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:09.954241 3309 ts_tablet_manager.cc:1905] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:09.955740 3308 ts_tablet_manager.cc:1905] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:09.955804 3309 ts_tablet_manager.cc:1918] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:09.955964 3309 log.cc:1196] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/42926cd5237544fe9de63424894f8ebc
I20260620 19:17:09.956437 3309 ts_tablet_manager.cc:1939] T 42926cd5237544fe9de63424894f8ebc P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting consensus metadata
I20260620 19:17:09.956822 2741 catalog_manager.cc:4841] TS e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079): tablet 42926cd5237544fe9de63424894f8ebc (table test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]) successfully deleted
I20260620 19:17:09.957372 3310 ts_tablet_manager.cc:1905] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:09.957428 3257 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: No bootstrap required, opened a new log
I20260620 19:17:09.957460 3257 ts_tablet_manager.cc:1397] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260620 19:17:09.957500 3308 ts_tablet_manager.cc:1918] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:09.957556 3308 log.cc:1196] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/42926cd5237544fe9de63424894f8ebc
I20260620 19:17:09.957574 3257 raft_consensus.cc:348] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.957630 3257 raft_consensus.cc:374] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.957647 3257 raft_consensus.cc:729] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.957693 3257 consensus_queue.cc:260] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.957778 3257 ts_tablet_manager.cc:1428] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.957783 3308 ts_tablet_manager.cc:1939] T 42926cd5237544fe9de63424894f8ebc P 128adcd646e9442f80bb3c831fdb380d: Deleting consensus metadata
I20260620 19:17:09.958254 2740 catalog_manager.cc:4841] TS 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279): tablet 42926cd5237544fe9de63424894f8ebc (table test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]) successfully deleted
I20260620 19:17:09.958864 3260 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: No bootstrap required, opened a new log
I20260620 19:17:09.958915 3260 ts_tablet_manager.cc:1397] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 19:17:09.959028 3260 raft_consensus.cc:348] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.959103 3260 raft_consensus.cc:374] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.959124 3260 raft_consensus.cc:729] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128adcd646e9442f80bb3c831fdb380d, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.959161 3260 consensus_queue.cc:260] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.959242 3260 ts_tablet_manager.cc:1428] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:09.959991 3262 tablet_bootstrap.cc:492] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: No bootstrap required, opened a new log
I20260620 19:17:09.960044 3262 ts_tablet_manager.cc:1397] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260620 19:17:09.960189 3262 raft_consensus.cc:348] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.960258 3262 raft_consensus.cc:374] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:09.960278 3262 raft_consensus.cc:729] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Initialized, Role: FOLLOWER
I20260620 19:17:09.960321 3262 consensus_queue.cc:260] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:09.960402 3262 ts_tablet_manager.cc:1428] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:09.964257 3310 ts_tablet_manager.cc:1918] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:09.964334 3310 log.cc:1196] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/42926cd5237544fe9de63424894f8ebc
I20260620 19:17:09.964562 3310 ts_tablet_manager.cc:1939] T 42926cd5237544fe9de63424894f8ebc P fc8801cf0b734771a6f3c436da852aa7: Deleting consensus metadata
I20260620 19:17:09.965025 2741 catalog_manager.cc:4841] TS fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043): tablet 42926cd5237544fe9de63424894f8ebc (table test-table [id=f2fbc01d081f4edb9ece9b9f049177a4]) successfully deleted
I20260620 19:17:10.034224 3302 raft_consensus.cc:482] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:10.034320 3302 raft_consensus.cc:504] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:10.034510 3302 leader_election.cc:288] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279), fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043)
I20260620 19:17:10.034694 3191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1e2b139064daca61ce5a61a18638a" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8801cf0b734771a6f3c436da852aa7" is_pre_election: true
I20260620 19:17:10.034693 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1e2b139064daca61ce5a61a18638a" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d" is_pre_election: true
I20260620 19:17:10.034788 3191 raft_consensus.cc:2454] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 0.
I20260620 19:17:10.034798 3051 raft_consensus.cc:2454] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 0.
I20260620 19:17:10.034924 2845 leader_election.cc:302] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d0c20cfea4447e8dfb6d7ec303482a, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:10.035017 3302 raft_consensus.cc:2781] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:10.035046 3302 raft_consensus.cc:482] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:10.035075 3302 raft_consensus.cc:3037] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:10.035694 3302 raft_consensus.cc:504] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:10.035815 3302 leader_election.cc:288] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 election: Requested vote from peers 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279), fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043)
I20260620 19:17:10.035989 3191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1e2b139064daca61ce5a61a18638a" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc8801cf0b734771a6f3c436da852aa7"
I20260620 19:17:10.035987 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ad1e2b139064daca61ce5a61a18638a" candidate_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d"
I20260620 19:17:10.036059 3051 raft_consensus.cc:3037] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:10.036058 3191 raft_consensus.cc:3037] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:10.036597 3191 raft_consensus.cc:2454] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 1.
I20260620 19:17:10.036757 2845 leader_election.cc:302] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d0c20cfea4447e8dfb6d7ec303482a, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:10.036832 3051 raft_consensus.cc:2454] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0d0c20cfea4447e8dfb6d7ec303482a in term 1.
I20260620 19:17:10.036864 3302 raft_consensus.cc:2781] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:10.036906 3302 raft_consensus.cc:686] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 LEADER]: Becoming Leader. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Running, Role: LEADER
I20260620 19:17:10.036957 3302 consensus_queue.cc:237] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:10.037551 2754 catalog_manager.cc:5495] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a reported cstate change: term changed from 0 to 1, leader changed from <none> to e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193). New cstate: current_term: 1 leader_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:10.051928 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:10.049837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.1.1.193:35079)
0620 19:17:10.049876 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:10.049878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:10.049885 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:10.049923 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:10.049924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:10.049933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:10.049978 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:10.050077 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.050515 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.051253 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.051458 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.051529 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:10.051693 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:10.051734 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:10.051792 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:10.051846 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:17:10.051861 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:10.055657 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:10.053691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60643 (local address 127.1.1.193:35079)
0620 19:17:10.053720 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:10.053721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:10.053728 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:10.053783 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:10.053783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:10.053789 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:10.053817 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:10.053914 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.054288 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.055170 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.055328 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.055377 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:10.055513 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:10.055542 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:10.055574 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:10.055611 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:17:10.055621 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:17:10.055728 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:10.053654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.1.1.193:60643)
0620 19:17:10.053695 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:10.053703 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:10.053730 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:10.053830 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:10.053830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:10.053831 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:17:10.053868 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:10.053869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.054348 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.054349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:10.055081 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:10.055084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:10.055393 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:10.055394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:10.055442 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:10.055449 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:17:10.055575 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:17:10.055576 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:17:10.055610 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 19:17:10.055648 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:17:10.057324 3051 raft_consensus.cc:1261] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Refusing update from remote peer e0d0c20cfea4447e8dfb6d7ec303482a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:10.057364 3191 raft_consensus.cc:1261] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Refusing update from remote peer e0d0c20cfea4447e8dfb6d7ec303482a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:10.057525 3306 consensus_queue.cc:1035] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Connected to new peer: Peer: permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:10.057721 3306 consensus_queue.cc:1035] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:17:10.061803 3286 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:17:11.581838 2752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:11.581926 2752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:11.582607 2752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Sending DeleteTablet for 3 replicas of tablet 5ad1e2b139064daca61ce5a61a18638a
I20260620 19:17:11.582824 2891 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad1e2b139064daca61ce5a61a18638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260620 19:17:11.582857 3171 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad1e2b139064daca61ce5a61a18638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44388
I20260620 19:17:11.582899 3031 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ad1e2b139064daca61ce5a61a18638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46700
I20260620 19:17:11.583069 3339 tablet_replica.cc:330] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: stopping tablet replica
I20260620 19:17:11.583276 3339 raft_consensus.cc:2229] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:11.583309 3339 raft_consensus.cc:2258] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.583506 3339 ts_tablet_manager.cc:1905] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:11.585124 3337 tablet_replica.cc:330] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: stopping tablet replica
I20260620 19:17:11.585186 3337 raft_consensus.cc:2229] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:11.585253 3339 ts_tablet_manager.cc:1918] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:17:11.585276 3337 raft_consensus.cc:2258] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.585294 3339 log.cc:1196] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/5ad1e2b139064daca61ce5a61a18638a
I20260620 19:17:11.585500 3339 ts_tablet_manager.cc:1939] T 5ad1e2b139064daca61ce5a61a18638a P 128adcd646e9442f80bb3c831fdb380d: Deleting consensus metadata
I20260620 19:17:11.585496 3337 ts_tablet_manager.cc:1905] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:11.585958 2740 catalog_manager.cc:4841] TS 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279): tablet 5ad1e2b139064daca61ce5a61a18638a (table test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]) successfully deleted
I20260620 19:17:11.586601 3338 tablet_replica.cc:330] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: stopping tablet replica
I20260620 19:17:11.586673 3338 raft_consensus.cc:2229] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:11.586714 3338 raft_consensus.cc:2258] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.586887 3337 ts_tablet_manager.cc:1918] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:17:11.586939 3337 log.cc:1196] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/5ad1e2b139064daca61ce5a61a18638a
I20260620 19:17:11.586920 3338 ts_tablet_manager.cc:1905] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:11.587152 3337 ts_tablet_manager.cc:1939] T 5ad1e2b139064daca61ce5a61a18638a P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting consensus metadata
I20260620 19:17:11.587411 2741 catalog_manager.cc:4841] TS e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079): tablet 5ad1e2b139064daca61ce5a61a18638a (table test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]) successfully deleted
I20260620 19:17:11.588228 3338 ts_tablet_manager.cc:1918] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:17:11.588294 3338 log.cc:1196] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/5ad1e2b139064daca61ce5a61a18638a
I20260620 19:17:11.588532 3338 ts_tablet_manager.cc:1939] T 5ad1e2b139064daca61ce5a61a18638a P fc8801cf0b734771a6f3c436da852aa7: Deleting consensus metadata
I20260620 19:17:11.588841 2741 catalog_manager.cc:4841] TS fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043): tablet 5ad1e2b139064daca61ce5a61a18638a (table test-table [id=2cbaaf7d976c49fc97fc284f1f66329d]) successfully deleted
Jun 20 19:17:11 dist-test-slave-mwkb krb5kdc[2709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:17:11 dist-test-slave-mwkb krb5kdc[2709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:11.605959 3347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:11.599532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46814 (local address 127.1.1.254:38845)
0620 19:17:11.599665 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:11.599668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:11.599680 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:17:11.599754 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:11.599754 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:11.599764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:11.599830 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:11.600290 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:11.600793 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:11.601595 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:11.601817 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:11.601876 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:11.603729 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:11.603730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:11.603731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:11.603736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:11.605199 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:11.605458 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:11.605458 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:11.605459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:11.605505 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:11.605637 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:11.605638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:11.605638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:11.605723 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:11.605784 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:11.605849 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:17:11.605901 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":86,"threads_started":1}
I20260620 19:17:11.607355 2752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:11.607499 2752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:11.608642 3031 tablet_service.cc:1467] Processing CreateTablet for tablet d6729a8d0dc1488981947ea9bbf4055c (DEFAULT_TABLE table=test-table [id=cac3001d03944d40a9af5855a7e6131f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.608666 3171 tablet_service.cc:1467] Processing CreateTablet for tablet d6729a8d0dc1488981947ea9bbf4055c (DEFAULT_TABLE table=test-table [id=cac3001d03944d40a9af5855a7e6131f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.608759 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6729a8d0dc1488981947ea9bbf4055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.608762 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6729a8d0dc1488981947ea9bbf4055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.609167 2891 tablet_service.cc:1467] Processing CreateTablet for tablet d6729a8d0dc1488981947ea9bbf4055c (DEFAULT_TABLE table=test-table [id=cac3001d03944d40a9af5855a7e6131f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.609282 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6729a8d0dc1488981947ea9bbf4055c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.610069 3348 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Bootstrap starting.
I20260620 19:17:11.610351 3348 tablet_bootstrap.cc:654] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.610438 3350 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Bootstrap starting.
I20260620 19:17:11.610625 3349 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Bootstrap starting.
I20260620 19:17:11.610814 3350 tablet_bootstrap.cc:654] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.610940 3349 tablet_bootstrap.cc:654] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.610944 3348 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: No bootstrap required, opened a new log
I20260620 19:17:11.611006 3348 ts_tablet_manager.cc:1397] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:11.611151 3348 raft_consensus.cc:348] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.611245 3348 raft_consensus.cc:374] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.611279 3348 raft_consensus.cc:729] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128adcd646e9442f80bb3c831fdb380d, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.611339 3350 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: No bootstrap required, opened a new log
I20260620 19:17:11.611337 3348 consensus_queue.cc:260] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.611390 3350 ts_tablet_manager.cc:1397] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:11.611514 3348 ts_tablet_manager.cc:1428] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:11.611507 3350 raft_consensus.cc:348] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.611564 3350 raft_consensus.cc:374] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.611583 3350 raft_consensus.cc:729] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.611634 3350 consensus_queue.cc:260] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.611745 3350 ts_tablet_manager.cc:1428] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:11.612092 3349 tablet_bootstrap.cc:492] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: No bootstrap required, opened a new log
I20260620 19:17:11.612141 3349 ts_tablet_manager.cc:1397] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:11.612252 3349 raft_consensus.cc:348] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.612320 3349 raft_consensus.cc:374] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.612344 3349 raft_consensus.cc:729] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.612387 3349 consensus_queue.cc:260] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.612542 3349 ts_tablet_manager.cc:1428] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:11.877334 3355 raft_consensus.cc:482] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:11.877430 3355 raft_consensus.cc:504] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.877599 3355 leader_election.cc:288] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:11.877789 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6729a8d0dc1488981947ea9bbf4055c" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" is_pre_election: true
I20260620 19:17:11.877799 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6729a8d0dc1488981947ea9bbf4055c" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d" is_pre_election: true
I20260620 19:17:11.877897 3051 raft_consensus.cc:2454] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:11.877898 2911 raft_consensus.cc:2454] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:11.878080 3125 leader_election.cc:302] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d0c20cfea4447e8dfb6d7ec303482a, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:11.878196 3355 raft_consensus.cc:2781] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:11.878229 3355 raft_consensus.cc:482] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:11.878257 3355 raft_consensus.cc:3037] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.878989 3355 raft_consensus.cc:504] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.879206 3355 leader_election.cc:288] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Requested vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:11.879271 3051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6729a8d0dc1488981947ea9bbf4055c" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d"
I20260620 19:17:11.879338 3051 raft_consensus.cc:3037] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.879354 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6729a8d0dc1488981947ea9bbf4055c" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
I20260620 19:17:11.879441 2911 raft_consensus.cc:3037] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.880023 3051 raft_consensus.cc:2454] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:11.880206 3124 leader_election.cc:302] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128adcd646e9442f80bb3c831fdb380d, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:11.880276 2911 raft_consensus.cc:2454] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:11.880299 3355 raft_consensus.cc:2781] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:11.880448 3355 raft_consensus.cc:686] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Becoming Leader. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Running, Role: LEADER
I20260620 19:17:11.880638 3355 consensus_queue.cc:237] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.881501 2752 catalog_manager.cc:5495] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "fc8801cf0b734771a6f3c436da852aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } health_report { overall_health: HEALTHY } } }
I20260620 19:17:11.895941 3359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:11.893427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.1.1.195:45043)
0620 19:17:11.893593 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:11.893595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:11.893600 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:17:11.893617 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:11.893618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:11.893626 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:11.893699 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:11.893880 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:11.894375 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:11.895204 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:11.895417 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:11.895499 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:11.895660 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:11.895694 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:11.895744 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:11.895805 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:17:11.895867 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 19:17:11.898074 2911 raft_consensus.cc:1261] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:11.898077 3051 raft_consensus.cc:1261] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:11.898257 3356 consensus_queue.cc:1035] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:11.898363 3355 consensus_queue.cc:1035] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:11.908221 2754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:11.908304 2754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:11.908905 2754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Sending DeleteTablet for 3 replicas of tablet d6729a8d0dc1488981947ea9bbf4055c
I20260620 19:17:11.909162 2891 tablet_service.cc:1514] Processing DeleteTablet for tablet d6729a8d0dc1488981947ea9bbf4055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260620 19:17:11.909181 3171 tablet_service.cc:1514] Processing DeleteTablet for tablet d6729a8d0dc1488981947ea9bbf4055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44388
I20260620 19:17:11.909191 3031 tablet_service.cc:1514] Processing DeleteTablet for tablet d6729a8d0dc1488981947ea9bbf4055c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:11 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46700
I20260620 19:17:11.909260 3337 tablet_replica.cc:330] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: stopping tablet replica
I20260620 19:17:11.909260 3338 tablet_replica.cc:330] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: stopping tablet replica
I20260620 19:17:11.909273 3339 tablet_replica.cc:330] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: stopping tablet replica
I20260620 19:17:11.909307 3337 raft_consensus.cc:2229] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:11.909309 3338 raft_consensus.cc:2229] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:11.909318 3339 raft_consensus.cc:2229] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:11.909346 3337 raft_consensus.cc:2258] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.909355 3339 raft_consensus.cc:2258] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.909365 3338 raft_consensus.cc:2258] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:11.909629 3339 ts_tablet_manager.cc:1905] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:11.909649 2754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:17:11.909729 3337 ts_tablet_manager.cc:1905] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:17:11.909765 2754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:11.910979 3031 tablet_service.cc:1467] Processing CreateTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 (DEFAULT_TABLE table=test-table [id=186773774aa047bdba86f9f1bbc166d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.911084 3031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e2c07984a741eeb75cd2bc46de2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.911180 3339 ts_tablet_manager.cc:1918] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:11.911222 3339 log.cc:1196] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/d6729a8d0dc1488981947ea9bbf4055c
I20260620 19:17:11.911237 3337 ts_tablet_manager.cc:1918] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:11.911276 3337 log.cc:1196] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/d6729a8d0dc1488981947ea9bbf4055c
I20260620 19:17:11.911435 3337 ts_tablet_manager.cc:1939] T d6729a8d0dc1488981947ea9bbf4055c P e0d0c20cfea4447e8dfb6d7ec303482a: Deleting consensus metadata
I20260620 19:17:11.911437 3339 ts_tablet_manager.cc:1939] T d6729a8d0dc1488981947ea9bbf4055c P 128adcd646e9442f80bb3c831fdb380d: Deleting consensus metadata
I20260620 19:17:11.911535 2891 tablet_service.cc:1467] Processing CreateTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 (DEFAULT_TABLE table=test-table [id=186773774aa047bdba86f9f1bbc166d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.911640 3171 tablet_service.cc:1467] Processing CreateTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 (DEFAULT_TABLE table=test-table [id=186773774aa047bdba86f9f1bbc166d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:11.911684 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e2c07984a741eeb75cd2bc46de2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.911752 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e2c07984a741eeb75cd2bc46de2789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:11.911805 2740 catalog_manager.cc:4841] TS 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279): tablet d6729a8d0dc1488981947ea9bbf4055c (table test-table [id=cac3001d03944d40a9af5855a7e6131f]) successfully deleted
I20260620 19:17:11.912210 3348 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Bootstrap starting.
I20260620 19:17:11.912817 3349 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Bootstrap starting.
I20260620 19:17:11.912823 3348 tablet_bootstrap.cc:654] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.912846 3350 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a: Bootstrap starting.
I20260620 19:17:11.913131 2741 catalog_manager.cc:4841] TS e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079): tablet d6729a8d0dc1488981947ea9bbf4055c (table test-table [id=cac3001d03944d40a9af5855a7e6131f]) successfully deleted
I20260620 19:17:11.913156 3349 tablet_bootstrap.cc:654] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.913166 3350 tablet_bootstrap.cc:654] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:11.913456 3338 ts_tablet_manager.cc:1905] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:11.913791 3350 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a: No bootstrap required, opened a new log
I20260620 19:17:11.913800 3348 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: No bootstrap required, opened a new log
I20260620 19:17:11.913841 3348 ts_tablet_manager.cc:1397] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:11.913841 3350 ts_tablet_manager.cc:1397] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:11.913944 3350 raft_consensus.cc:348] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.914001 3350 raft_consensus.cc:374] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.913965 3348 raft_consensus.cc:348] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.914021 3350 raft_consensus.cc:729] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0d0c20cfea4447e8dfb6d7ec303482a, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.914023 3348 raft_consensus.cc:374] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.914044 3348 raft_consensus.cc:729] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128adcd646e9442f80bb3c831fdb380d, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.914073 3350 consensus_queue.cc:260] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.914085 3348 consensus_queue.cc:260] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.914162 3350 ts_tablet_manager.cc:1428] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:11.914176 3348 ts_tablet_manager.cc:1428] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:11.914800 3338 ts_tablet_manager.cc:1918] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:17:11.914845 3338 log.cc:1196] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/d6729a8d0dc1488981947ea9bbf4055c
I20260620 19:17:11.914991 3338 ts_tablet_manager.cc:1939] T d6729a8d0dc1488981947ea9bbf4055c P fc8801cf0b734771a6f3c436da852aa7: Deleting consensus metadata
I20260620 19:17:11.915411 2741 catalog_manager.cc:4841] TS fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043): tablet d6729a8d0dc1488981947ea9bbf4055c (table test-table [id=cac3001d03944d40a9af5855a7e6131f]) successfully deleted
I20260620 19:17:11.915519 3349 tablet_bootstrap.cc:492] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: No bootstrap required, opened a new log
I20260620 19:17:11.915570 3349 ts_tablet_manager.cc:1397] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:17:11.915670 3349 raft_consensus.cc:348] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.915745 3349 raft_consensus.cc:374] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:11.915771 3349 raft_consensus.cc:729] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Initialized, Role: FOLLOWER
I20260620 19:17:11.915824 3349 consensus_queue.cc:260] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.915921 3349 ts_tablet_manager.cc:1428] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:11.952052 3355 raft_consensus.cc:482] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:11.952119 3355 raft_consensus.cc:504] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.952284 3355 leader_election.cc:288] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:11.952419 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e2c07984a741eeb75cd2bc46de2789" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" is_pre_election: true
I20260620 19:17:11.952464 3040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e2c07984a741eeb75cd2bc46de2789" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d" is_pre_election: true
I20260620 19:17:11.952514 2911 raft_consensus.cc:2454] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:11.952535 3040 raft_consensus.cc:2454] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 0.
I20260620 19:17:11.952708 3125 leader_election.cc:302] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e0d0c20cfea4447e8dfb6d7ec303482a, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:11.952804 3355 raft_consensus.cc:2781] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:11.952862 3355 raft_consensus.cc:482] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:11.952894 3355 raft_consensus.cc:3037] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.953568 3355 raft_consensus.cc:504] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.953724 3355 leader_election.cc:288] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Requested vote from peers e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079), 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279)
I20260620 19:17:11.953840 2911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e2c07984a741eeb75cd2bc46de2789" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a"
I20260620 19:17:11.953853 3040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e2c07984a741eeb75cd2bc46de2789" candidate_uuid: "fc8801cf0b734771a6f3c436da852aa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128adcd646e9442f80bb3c831fdb380d"
I20260620 19:17:11.953912 3040 raft_consensus.cc:3037] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.953912 2911 raft_consensus.cc:3037] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:11.954713 2911 raft_consensus.cc:2454] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:11.954749 3040 raft_consensus.cc:2454] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc8801cf0b734771a6f3c436da852aa7 in term 1.
I20260620 19:17:11.954895 3124 leader_election.cc:302] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 128adcd646e9442f80bb3c831fdb380d, fc8801cf0b734771a6f3c436da852aa7; no voters:
I20260620 19:17:11.955049 3355 raft_consensus.cc:2781] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:11.955104 3355 raft_consensus.cc:686] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Becoming Leader. State: Replica: fc8801cf0b734771a6f3c436da852aa7, State: Running, Role: LEADER
I20260620 19:17:11.955154 3355 consensus_queue.cc:237] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } }
I20260620 19:17:11.955732 2754 catalog_manager.cc:5495] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "fc8801cf0b734771a6f3c436da852aa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc8801cf0b734771a6f3c436da852aa7" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45043 } health_report { overall_health: HEALTHY } } }
I20260620 19:17:11.961458 2911 raft_consensus.cc:1261] T 54e2c07984a741eeb75cd2bc46de2789 P e0d0c20cfea4447e8dfb6d7ec303482a [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:11.961499 3040 raft_consensus.cc:1261] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Refusing update from remote peer fc8801cf0b734771a6f3c436da852aa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:11.961675 3363 consensus_queue.cc:1035] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0d0c20cfea4447e8dfb6d7ec303482a" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 35079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:11.961755 3355 consensus_queue.cc:1035] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "128adcd646e9442f80bb3c831fdb380d" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:17:11.965426 3321 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:17:13.480790 2753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:13.480883 2753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:13.481666 2753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65aa39f8a7a646d697f0e9735051e6ee: Sending DeleteTablet for 3 replicas of tablet 54e2c07984a741eeb75cd2bc46de2789
I20260620 19:17:13.481922 3031 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46700
I20260620 19:17:13.481945 2891 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:38214
I20260620 19:17:13.482077 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2826
I20260620 19:17:13.482116 3391 tablet_replica.cc:330] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: stopping tablet replica
I20260620 19:17:13.482126 3171 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e2c07984a741eeb75cd2bc46de2789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44388
I20260620 19:17:13.482178 3391 raft_consensus.cc:2229] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:13.483325 3391 raft_consensus.cc:2258] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:13.483443 3393 tablet_replica.cc:330] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: stopping tablet replica
I20260620 19:17:13.483489 3393 raft_consensus.cc:2229] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:13.483536 3393 raft_consensus.cc:2258] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:13.483608 3391 ts_tablet_manager.cc:1905] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:13.483726 3393 ts_tablet_manager.cc:1905] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:13.485177 3391 ts_tablet_manager.cc:1918] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:17:13.485222 3391 log.cc:1196] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/54e2c07984a741eeb75cd2bc46de2789
I20260620 19:17:13.485376 3391 ts_tablet_manager.cc:1939] T 54e2c07984a741eeb75cd2bc46de2789 P 128adcd646e9442f80bb3c831fdb380d: Deleting consensus metadata
I20260620 19:17:13.485581 3393 ts_tablet_manager.cc:1918] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:17:13.485631 3393 log.cc:1196] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/54e2c07984a741eeb75cd2bc46de2789
I20260620 19:17:13.485616 2740 catalog_manager.cc:4841] TS 128adcd646e9442f80bb3c831fdb380d (127.1.1.194:37279): tablet 54e2c07984a741eeb75cd2bc46de2789 (table test-table [id=186773774aa047bdba86f9f1bbc166d5]) successfully deleted
I20260620 19:17:13.485872 3393 ts_tablet_manager.cc:1939] T 54e2c07984a741eeb75cd2bc46de2789 P fc8801cf0b734771a6f3c436da852aa7: Deleting consensus metadata
I20260620 19:17:13.486227 2741 catalog_manager.cc:4841] TS fc8801cf0b734771a6f3c436da852aa7 (127.1.1.195:45043): tablet 54e2c07984a741eeb75cd2bc46de2789 (table test-table [id=186773774aa047bdba86f9f1bbc166d5]) successfully deleted
W20260620 19:17:13.489110 2741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:35079 (error 108)
I20260620 19:17:13.489473 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2966
I20260620 19:17:13.490190 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:13.489506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:35079 (local address 127.0.0.1:44186)
0620 19:17:13.490098 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:13.490146 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:35079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":67,"threads_started":1}
W20260620 19:17:13.490273 2741 catalog_manager.cc:4584] TS e0d0c20cfea4447e8dfb6d7ec303482a (127.1.1.193:35079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54e2c07984a741eeb75cd2bc46de2789: Network error: Client connection negotiation failed: client connection to 127.1.1.193:35079: connect: Connection refused (error 111)
I20260620 19:17:13.494321 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3106
I20260620 19:17:13.499368 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 2722
2026-06-20T19:17:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5922 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:13 dist-test-slave-mwkb krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:13 dist-test-slave-mwkb krb5kdc[3398](info): set up 2 sockets
Jun 20 19:17:13 dist-test-slave-mwkb krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260620 19:17:14.448405 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:14.548513 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34183
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:34183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:14.618508 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:14.618702 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:14.618733 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:14.619724 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:14.619786 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:14.619809 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:14.619838 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:14.619867 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:14.621310 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:34183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:14.621565 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:14.621770 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:14.626168 3411 server_base.cc:1034] running on GCE node
W20260620 19:17:14.626116 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:14.626122 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:14.626119 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:14.626554 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:14.626832 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:14.627995 3411 hybrid_clock.cc:648] HybridClock initialized: now 1781983034627982 us; error 45 us; skew 500 ppm
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:14.630323 3411 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:14.630632 3411 webserver.cc:460] Webserver started at http://127.1.1.254:37555/ using document root <none> and password file <none>
I20260620 19:17:14.630849 3411 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:14.630916 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:14.631060 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:14.632061 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "f6412d94b1434e43bca361c2754fb179"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "2ba6e64afca4b221d9f5bd3e82c7f9ef"
server_key_iv: "a5f0f66d4f7538e99d804a1eb12b4646"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.632371 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "f6412d94b1434e43bca361c2754fb179"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "2ba6e64afca4b221d9f5bd3e82c7f9ef"
server_key_iv: "a5f0f66d4f7538e99d804a1eb12b4646"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.633756 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:17:14.634577 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.634742 3411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:14.634838 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "f6412d94b1434e43bca361c2754fb179"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "2ba6e64afca4b221d9f5bd3e82c7f9ef"
server_key_iv: "a5f0f66d4f7538e99d804a1eb12b4646"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.634902 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:14.647473 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:14.649442 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:14.649588 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:14.652895 3411 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:34183
I20260620 19:17:14.652941 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:34183 every 8 connection(s)
I20260620 19:17:14.653259 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:14.653980 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:14.655866 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179: Bootstrap starting.
I20260620 19:17:14.656366 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:14.656631 3479 log.cc:824] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:14.657231 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179: No bootstrap required, opened a new log
I20260620 19:17:14.658533 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } }
I20260620 19:17:14.658661 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:14.658716 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6412d94b1434e43bca361c2754fb179, State: Initialized, Role: FOLLOWER
I20260620 19:17:14.658901 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } }
I20260620 19:17:14.658990 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:14.659030 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:14.659075 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:14.659519 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } }
I20260620 19:17:14.659652 3479 leader_election.cc:302] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6412d94b1434e43bca361c2754fb179; no voters:
I20260620 19:17:14.659819 3479 leader_election.cc:288] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:14.659850 3482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:14.660048 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:14.660190 3482 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [term 1 LEADER]: Becoming Leader. State: Replica: f6412d94b1434e43bca361c2754fb179, State: Running, Role: LEADER
I20260620 19:17:14.660288 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } }
I20260620 19:17:14.660650 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6412d94b1434e43bca361c2754fb179. Latest consensus state: current_term: 1 leader_uuid: "f6412d94b1434e43bca361c2754fb179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } } }
I20260620 19:17:14.660656 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6412d94b1434e43bca361c2754fb179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6412d94b1434e43bca361c2754fb179" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34183 } } }
I20260620 19:17:14.660748 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:14.660758 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:14.660949 3491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:14.661484 3491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:14.661880 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3411
I20260620 19:17:14.661952 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:14.662163 1031 external_mini_cluster.cc:1428] Setting key 018ccc60d68e980bf3df9714a8edd3c5
I20260620 19:17:14.663720 3491 catalog_manager.cc:1347] Generated new cluster ID: 363ee39338344c94b40d1c2d9cd00b32
I20260620 19:17:14.663765 3491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:14.669198 3491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:14.669833 3491 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:14.675040 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:14.662713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.1.1.254:34183)
0620 19:17:14.665197 (+ 2484us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:14.665205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:14.665234 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:17:14.665335 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:14.665341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:14.665390 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:14.665519 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:14.666153 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.667517 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.668447 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.669064 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.669179 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:14.671785 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:14.671790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:14.671790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:14.671802 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:14.673546 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.673922 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.673925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.673926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.674025 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.674223 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.674224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.674224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.674471 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:14.674596 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:14.674720 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 19:17:14.674843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2440,"thread_start_us":71,"threads_started":1}
I20260620 19:17:14.677438 3491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6412d94b1434e43bca361c2754fb179: Generated new TSK 0
I20260620 19:17:14.677572 3491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:14.701555 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:14.775460 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:14.775624 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:14.775645 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:14.776631 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:14.776666 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:14.776690 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:14.777956 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:14.778170 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:14.778373 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:14.782977 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:14.782967 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:14.783015 3505 server_base.cc:1034] running on GCE node
W20260620 19:17:14.782967 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:14.783340 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:14.783572 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:14.784737 3505 hybrid_clock.cc:648] HybridClock initialized: now 1781983034784724 us; error 24 us; skew 500 ppm
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:14.790972 3505 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:14.791371 3505 webserver.cc:460] Webserver started at http://127.1.1.193:40741/ using document root <none> and password file <none>
I20260620 19:17:14.791599 3505 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:14.791664 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:14.791806 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:14.792757 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "7ec9fd0df704448392c26db1e7f1ea00"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "d46031c81dde0fa699430f52539b5da5"
server_key_iv: "664fefb6c46b8f39915a699f5b94eb85"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.793170 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "7ec9fd0df704448392c26db1e7f1ea00"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "d46031c81dde0fa699430f52539b5da5"
server_key_iv: "664fefb6c46b8f39915a699f5b94eb85"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.794437 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:14.795053 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.795214 3505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:14.795285 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "7ec9fd0df704448392c26db1e7f1ea00"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "d46031c81dde0fa699430f52539b5da5"
server_key_iv: "664fefb6c46b8f39915a699f5b94eb85"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.795336 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:14.824970 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:14.826800 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:14.826944 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:14.827126 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:14.827370 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:14.827417 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.827447 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:14.827469 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.832583 3505 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:33811
I20260620 19:17:14.832618 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:33811 every 8 connection(s)
I20260620 19:17:14.832880 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:14.835544 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3505
I20260620 19:17:14.835618 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:14.835773 1031 external_mini_cluster.cc:1428] Setting key fe4a1be237f4258cb369257879b1778f
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:14.841495 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:14.833521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:56031 (local address 127.1.1.254:34183)
0620 19:17:14.833571 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:14.833572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:14.833743 (+ 171us) server_negotiation.cc:407] Connection header received
0620 19:17:14.834140 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:14.834141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:14.834151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:14.834198 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:14.834917 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.835357 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.836343 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.836543 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.836592 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:14.838682 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:14.838683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:14.838683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:14.838689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:14.840060 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.840526 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.840527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.840527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.840573 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.840869 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.840869 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.840869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.840961 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:14.841006 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:14.841372 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 19:17:14.841436 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:17:14.841765 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:14.833493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34183 (local address 127.1.1.193:56031)
0620 19:17:14.833638 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:14.833672 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:14.834064 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:14.834240 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:14.834243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:14.834698 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:14.834852 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:14.834856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.835451 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.835452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:14.836272 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:14.836274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.836619 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.836620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:14.836747 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:14.837146 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:14.837154 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:14.838604 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:14.840141 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:14.840148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:14.840150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:14.840450 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:14.840638 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:14.840639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:14.840639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:14.840780 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:14.841023 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:14.841029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:14.841223 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 19:17:14.841342 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:17:14.841514 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260620 19:17:14.842275 3634 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34183
I20260620 19:17:14.842355 3634 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:14.842530 3634 heartbeater.cc:511] Master 127.1.1.254:34183 requested a full tablet report, sending...
I20260620 19:17:14.842993 3443 ts_manager.cc:194] Registered new tserver with Master: 7ec9fd0df704448392c26db1e7f1ea00 (127.1.1.193:33811)
I20260620 19:17:14.843513 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:56031
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:14.859675 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:14.935899 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:14.936123 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:14.936161 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:14.937320 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:14.937384 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:14.937429 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:14.938889 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:14.939139 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:14.939361 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:14.944052 3639 server_base.cc:1034] running on GCE node
W20260620 19:17:14.943962 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:14.943976 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:14.943964 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:14.944366 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:14.944620 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:14.945755 3639 hybrid_clock.cc:648] HybridClock initialized: now 1781983034945737 us; error 33 us; skew 500 ppm
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:14.948081 3639 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:14.948491 3639 webserver.cc:460] Webserver started at http://127.1.1.194:43361/ using document root <none> and password file <none>
I20260620 19:17:14.948711 3639 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:14.948753 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:14.948904 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:14.949908 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "d137562934f24e7981e086509967c08b"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "0611eab62c085c006b166c11bb5342e4"
server_key_iv: "636e9e471a6feeb349046e741d90e758"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.950232 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "d137562934f24e7981e086509967c08b"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "0611eab62c085c006b166c11bb5342e4"
server_key_iv: "636e9e471a6feeb349046e741d90e758"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.951427 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:14.952118 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.952286 3639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:14.952365 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "d137562934f24e7981e086509967c08b"
format_stamp: "Formatted at 2026-06-20 19:17:14 on dist-test-slave-mwkb"
server_key: "0611eab62c085c006b166c11bb5342e4"
server_key_iv: "636e9e471a6feeb349046e741d90e758"
server_key_version: "encryptionkey@0"
I20260620 19:17:14.952414 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:14.963001 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:14.964488 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:14.964581 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:14.964725 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:14.964979 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:14.965029 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.965062 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:14.965104 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:14.970121 3639 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:37177
I20260620 19:17:14.970189 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:37177 every 8 connection(s)
I20260620 19:17:14.970443 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:14.973742 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3639
I20260620 19:17:14.973850 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:14.974043 1031 external_mini_cluster.cc:1428] Setting key 2c3bc09c0622762a413c463b917968ce
Jun 20 19:17:14 dist-test-slave-mwkb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983034, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:14.979308 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:14.971115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:60481 (local address 127.1.1.254:34183)
0620 19:17:14.971161 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:14.971163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:14.971368 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:17:14.971749 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:14.971750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:14.971759 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:14.971808 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:14.972594 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.973036 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.974065 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.974376 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.974447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:14.976638 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:14.976639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:14.976639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:14.976645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:14.977930 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.978416 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.978417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.978418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.978469 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:14.978735 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:14.978736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:14.978736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:14.978826 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:14.978873 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:14.979198 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 19:17:14.979253 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:14.979610 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:14.971100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34183 (local address 127.1.1.194:60481)
0620 19:17:14.971259 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:14.971289 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:14.971676 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:14.971876 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:14.971879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:14.972366 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:14.972521 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:14.972524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.973136 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.973136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:14.973978 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:14.973980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:14.974465 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:14.974466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:14.974593 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:14.975078 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:14.975085 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:14.976524 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:14.978017 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:14.978022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:14.978024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:14.978342 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:14.978533 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:14.978534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:14.978534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:14.978667 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:14.978889 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:14.978894 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:14.979043 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 19:17:14.979170 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 19:17:14.979336 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260620 19:17:14.980087 3768 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34183
I20260620 19:17:14.980182 3768 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:14.980356 3768 heartbeater.cc:511] Master 127.1.1.254:34183 requested a full tablet report, sending...
I20260620 19:17:14.980685 3443 ts_manager.cc:194] Registered new tserver with Master: d137562934f24e7981e086509967c08b (127.1.1.194:37177)
I20260620 19:17:14.981037 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:60481
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:14.996896 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:15.069832 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:15.069993 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:15.070012 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:15.071094 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:15.071134 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:15.071187 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:15.073467 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:15.073693 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:15.073906 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:15.078115 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:15.078183 3773 server_base.cc:1034] running on GCE node
W20260620 19:17:15.078251 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:15.078322 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:15.078490 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:15.078727 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:15.079881 3773 hybrid_clock.cc:648] HybridClock initialized: now 1781983035079852 us; error 54 us; skew 500 ppm
Jun 20 19:17:15 dist-test-slave-mwkb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:15.082245 3773 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:15.082640 3773 webserver.cc:460] Webserver started at http://127.1.1.195:42917/ using document root <none> and password file <none>
I20260620 19:17:15.082826 3773 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:15.082885 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:15.083014 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:15.083861 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "707561fef59140b29cdc59b599118622"
format_stamp: "Formatted at 2026-06-20 19:17:15 on dist-test-slave-mwkb"
server_key: "df462be850a376be5688306a408f866f"
server_key_iv: "5166bb99c8a30bba35aa3b36037897b0"
server_key_version: "encryptionkey@0"
I20260620 19:17:15.084164 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "707561fef59140b29cdc59b599118622"
format_stamp: "Formatted at 2026-06-20 19:17:15 on dist-test-slave-mwkb"
server_key: "df462be850a376be5688306a408f866f"
server_key_iv: "5166bb99c8a30bba35aa3b36037897b0"
server_key_version: "encryptionkey@0"
I20260620 19:17:15.085282 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:15.085866 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:15.085999 3773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:15.086084 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "707561fef59140b29cdc59b599118622"
format_stamp: "Formatted at 2026-06-20 19:17:15 on dist-test-slave-mwkb"
server_key: "df462be850a376be5688306a408f866f"
server_key_iv: "5166bb99c8a30bba35aa3b36037897b0"
server_key_version: "encryptionkey@0"
I20260620 19:17:15.086161 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:15.111253 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:15.113341 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:15.113490 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:15.113684 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:15.113924 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:15.113969 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:15.114003 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:15.114027 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:15.119275 3773 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:40079
I20260620 19:17:15.119319 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:40079 every 8 connection(s)
I20260620 19:17:15.119596 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:15.120502 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3773
I20260620 19:17:15.120607 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:15.120765 1031 external_mini_cluster.cc:1428] Setting key f56c01c27a895c947ca21a406aa5ac45
Jun 20 19:17:15 dist-test-slave-mwkb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983035, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:15.127333 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:15.120253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40857 (local address 127.1.1.254:34183)
0620 19:17:15.120314 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:15.120316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:15.120480 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:17:15.121045 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:15.121048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:15.121058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:15.121158 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:15.122121 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:15.122579 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:15.123415 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:15.123591 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:15.123637 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:15.125335 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:15.125337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:15.125337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:15.125343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:15.126379 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:15.126703 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:15.126704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:15.126704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:15.126746 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:15.126920 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:15.126920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:15.126920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:15.127010 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:15.127051 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:15.127223 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:17:15.127278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:17:15.127434 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:15.120215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34183 (local address 127.1.1.195:40857)
0620 19:17:15.120351 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:15.120386 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:15.120946 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:15.121224 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:15.121228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:15.121852 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:15.122038 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:15.122042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:15.122646 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:15.122647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:15.123345 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:15.123358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:15.123653 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:15.123653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:15.123734 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:15.124063 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:15.124070 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:15.125267 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:15.126440 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:15.126445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:15.126446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:15.126641 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:15.126796 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:15.126796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:15.126796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:15.126875 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:15.127050 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:15.127053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:15.127137 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 19:17:15.127201 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:17:15.127291 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260620 19:17:15.127807 3902 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34183
I20260620 19:17:15.127892 3902 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:15.128026 3902 heartbeater.cc:511] Master 127.1.1.254:34183 requested a full tablet report, sending...
I20260620 19:17:15.128314 3443 ts_manager.cc:194] Registered new tserver with Master: 707561fef59140b29cdc59b599118622 (127.1.1.195:40079)
I20260620 19:17:15.128625 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:40857
I20260620 19:17:15.132350 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:17:15.142472 3501 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:15.141725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.1.1.254:34183)
0620 19:17:15.141789 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:15.141791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:15.141872 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:17:15.141906 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:15.141907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:15.141915 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:15.141960 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:15.142415 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39416: BlockingRecv error: recv got EOF from 127.0.0.1:39416 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:17:15.143252 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3505
I20260620 19:17:15.147207 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3639
I20260620 19:17:15.151265 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3773
I20260620 19:17:15.155244 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3411
2026-06-20T19:17:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1655 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:15 dist-test-slave-mwkb krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:15 dist-test-slave-mwkb krb5kdc[3914](info): set up 2 sockets
Jun 20 19:17:15 dist-test-slave-mwkb krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260620 19:17:16.112685 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:16.213469 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46059
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:16.289393 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:16.289567 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:16.289594 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:16.290673 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:16.290710 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:16.290724 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:16.290773 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:16.290800 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:16.292259 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:16.292520 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:16.292737 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:16.297367 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:16.297437 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:16.297454 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:16.297502 3927 server_base.cc:1034] running on GCE node
I20260620 19:17:16.297827 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:16.298106 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:16.299244 3927 hybrid_clock.cc:648] HybridClock initialized: now 1781983036299236 us; error 32 us; skew 500 ppm
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983036, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:16.301625 3927 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:16.302052 3927 webserver.cc:460] Webserver started at http://127.1.1.254:44817/ using document root <none> and password file <none>
I20260620 19:17:16.302277 3927 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:16.302345 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:16.302488 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:16.303442 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "06fa6de988324851b20af2632ae8d251"
format_stamp: "Formatted at 2026-06-20 19:17:16 on dist-test-slave-mwkb"
server_key: "eeee64cffdcb57f5157615f3de362a44"
server_key_iv: "5bb0382cbece179a78527b571c0484c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:16.303771 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "06fa6de988324851b20af2632ae8d251"
format_stamp: "Formatted at 2026-06-20 19:17:16 on dist-test-slave-mwkb"
server_key: "eeee64cffdcb57f5157615f3de362a44"
server_key_iv: "5bb0382cbece179a78527b571c0484c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:16.304988 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:16.305693 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:16.305882 3927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:16.305981 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "06fa6de988324851b20af2632ae8d251"
format_stamp: "Formatted at 2026-06-20 19:17:16 on dist-test-slave-mwkb"
server_key: "eeee64cffdcb57f5157615f3de362a44"
server_key_iv: "5bb0382cbece179a78527b571c0484c8"
server_key_version: "encryptionkey@0"
I20260620 19:17:16.306057 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:16.322469 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:16.324328 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:16.324465 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:16.327931 3927 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46059
I20260620 19:17:16.327975 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46059 every 8 connection(s)
I20260620 19:17:16.328263 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:16.328918 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:16.330687 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251: Bootstrap starting.
I20260620 19:17:16.331182 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:16.331411 3995 log.cc:824] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:16.331964 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251: No bootstrap required, opened a new log
I20260620 19:17:16.333315 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } }
I20260620 19:17:16.333427 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:16.333447 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06fa6de988324851b20af2632ae8d251, State: Initialized, Role: FOLLOWER
I20260620 19:17:16.333571 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } }
I20260620 19:17:16.333642 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:16.333686 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:16.333717 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:16.334386 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } }
I20260620 19:17:16.334465 3995 leader_election.cc:302] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06fa6de988324851b20af2632ae8d251; no voters:
I20260620 19:17:16.334578 3995 leader_election.cc:288] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:16.334642 3998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:16.334789 3998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [term 1 LEADER]: Becoming Leader. State: Replica: 06fa6de988324851b20af2632ae8d251, State: Running, Role: LEADER
I20260620 19:17:16.334787 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:16.334882 3998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } }
I20260620 19:17:16.335232 4000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06fa6de988324851b20af2632ae8d251. Latest consensus state: current_term: 1 leader_uuid: "06fa6de988324851b20af2632ae8d251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } } }
I20260620 19:17:16.335325 4000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:16.335229 3999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06fa6de988324851b20af2632ae8d251" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06fa6de988324851b20af2632ae8d251" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46059 } } }
I20260620 19:17:16.335453 3999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:16.335523 4006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:16.335971 4006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:16.337165 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 3927
I20260620 19:17:16.337255 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:16.337276 4006 catalog_manager.cc:1347] Generated new cluster ID: 9877ca31b0cf4d7191a9528318ddad32
I20260620 19:17:16.337307 4006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:16.337400 1031 external_mini_cluster.cc:1428] Setting key c4c44ee5d7e17ddf3f5c3fd9f41c006e
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:16.345016 4006 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:16.345649 4006 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:16.347533 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:16.337736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.1.1.254:46059)
0620 19:17:16.337854 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:16.337863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:16.337892 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:17:16.337989 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:16.337995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:16.338050 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:16.338168 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:16.338733 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:16.340079 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:16.341093 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:16.341811 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:16.341940 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:16.344294 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:16.344298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:16.344298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:16.344308 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:16.346115 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:16.346491 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:16.346493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:16.346494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:16.346577 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:16.346773 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:16.346773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:16.346774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:16.346994 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:16.347119 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:16.347226 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 19:17:16.347333 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":53,"threads_started":1}
I20260620 19:17:16.348390 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:17:16.365911 4006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06fa6de988324851b20af2632ae8d251: Generated new TSK 0
I20260620 19:17:16.366088 4006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:17:16.369500 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:16.366139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58932 (local address 127.1.1.254:46059)
0620 19:17:16.366209 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:16.366211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:16.366221 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:17:16.366274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:16.366275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:16.366284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:16.366342 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:16.366702 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:16.367338 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:16.368300 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:16.368478 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:16.368524 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:16.369130 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:16.369132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:16.369132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:16.369291 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:16.369303 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:16.369308 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:16.369353 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:16.369437 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:17:16.369456 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:17:16.371614 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 3927
2026-06-20T19:17:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1213 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[4028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[4028](info): set up 2 sockets
Jun 20 19:17:16 dist-test-slave-mwkb krb5kdc[4028](info): commencing operation
krb5kdc: starting...
W20260620 19:17:17.336436 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:17.435451 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33943
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:17.506858 4041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:17.507014 4041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:17.507031 4041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:17.508039 4041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:17.508069 4041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:17.508081 4041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:17.508093 4041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:17.508103 4041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:17.509370 4041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:33943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:17.509534 4041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:17.509672 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:17.513947 4047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.514005 4049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.514029 4041 server_base.cc:1034] running on GCE node
W20260620 19:17:17.513947 4046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.514374 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:17.514586 4041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:17.515714 4041 hybrid_clock.cc:648] HybridClock initialized: now 1781983037515705 us; error 24 us; skew 500 ppm
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:17.517711 4041 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:17.518075 4041 webserver.cc:460] Webserver started at http://127.1.1.254:34489/ using document root <none> and password file <none>
I20260620 19:17:17.518234 4041 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:17.518267 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:17.518337 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:17.519102 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "f1e32efe42cd439e88c993831469287a"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "f968fb856b996e9945d581613c38116d"
server_key_iv: "90a83befdfeab394bd1aac448fc722fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.519342 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "f1e32efe42cd439e88c993831469287a"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "f968fb856b996e9945d581613c38116d"
server_key_iv: "90a83befdfeab394bd1aac448fc722fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.520574 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:17.521312 4056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.521514 4041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:17:17.521620 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "f1e32efe42cd439e88c993831469287a"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "f968fb856b996e9945d581613c38116d"
server_key_iv: "90a83befdfeab394bd1aac448fc722fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.521701 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:17.530666 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:17.532346 4041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:17.532491 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:17.536187 4041 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:33943
I20260620 19:17:17.536213 4108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:33943 every 8 connection(s)
I20260620 19:17:17.536494 4041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:17.537359 4109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:17.539162 4109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a: Bootstrap starting.
I20260620 19:17:17.539235 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4041
I20260620 19:17:17.539341 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:17.539507 1031 external_mini_cluster.cc:1428] Setting key d342d1af41b344b36fffab4b16123b47
I20260620 19:17:17.539824 4109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:17.540391 4109 log.cc:824] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:17.541133 4109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a: No bootstrap required, opened a new log
I20260620 19:17:17.543323 4109 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } }
I20260620 19:17:17.543433 4109 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:17.543453 4109 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1e32efe42cd439e88c993831469287a, State: Initialized, Role: FOLLOWER
I20260620 19:17:17.543572 4109 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } }
I20260620 19:17:17.543668 4109 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:17.543706 4109 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:17.543753 4109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:17.544224 4109 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } }
I20260620 19:17:17.544330 4109 leader_election.cc:302] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1e32efe42cd439e88c993831469287a; no voters:
I20260620 19:17:17.544476 4109 leader_election.cc:288] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:17.544504 4114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:17.544605 4114 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [term 1 LEADER]: Becoming Leader. State: Replica: f1e32efe42cd439e88c993831469287a, State: Running, Role: LEADER
I20260620 19:17:17.544699 4109 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:17.544724 4114 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } }
I20260620 19:17:17.545056 4115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1e32efe42cd439e88c993831469287a. Latest consensus state: current_term: 1 leader_uuid: "f1e32efe42cd439e88c993831469287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } } }
I20260620 19:17:17.545200 4115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:17.545576 4123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:17.546089 4123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:17.546383 4116 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1e32efe42cd439e88c993831469287a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1e32efe42cd439e88c993831469287a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33943 } } }
I20260620 19:17:17.546480 4116 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:17.547642 4123 catalog_manager.cc:1347] Generated new cluster ID: f01d8d9d8b364c76ad190f8e06a5c98e
I20260620 19:17:17.547708 4123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:17.550031 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:17.540011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.1.1.254:33943)
0620 19:17:17.540195 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:17.540205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:17.540241 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:17:17.540790 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:17.540799 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:17.540858 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:17.540998 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:17.541642 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.543024 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.543872 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.544241 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.544347 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:17.546864 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:17.546867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:17.546867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:17.546875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:17.548555 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.548937 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.548939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.548939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.549025 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.549231 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.549232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.549232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.549490 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:17.549638 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:17.549744 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:17:17.549854 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":137,"thread_start_us":101,"threads_started":1}
I20260620 19:17:17.574282 4123 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:17.574780 4123 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:17.578616 4123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1e32efe42cd439e88c993831469287a: Generated new TSK 0
I20260620 19:17:17.578773 4123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:17.613787 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:17.703379 4135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:17.703572 4135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:17.703605 4135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:17.704638 4135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:17.704697 4135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:17.704739 4135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:17.706166 4135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:17.706409 4135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:17.706617 4135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:17.711380 4141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.711383 4143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.711510 4140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.711596 4135 server_base.cc:1034] running on GCE node
I20260620 19:17:17.711795 4135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:17.711978 4135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:17.713132 4135 hybrid_clock.cc:648] HybridClock initialized: now 1781983037713117 us; error 47 us; skew 500 ppm
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:17.715852 4135 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:17.716207 4135 webserver.cc:460] Webserver started at http://127.1.1.193:40807/ using document root <none> and password file <none>
I20260620 19:17:17.716390 4135 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:17.716428 4135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:17.716518 4135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:17.717499 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "3624a1a630034194b1f044dc6d2762fa"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "40a99177e5654ac5a8b156c3985c19f8"
server_key_iv: "feeaaa08a674f883716b5395371d52a7"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.717782 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "3624a1a630034194b1f044dc6d2762fa"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "40a99177e5654ac5a8b156c3985c19f8"
server_key_iv: "feeaaa08a674f883716b5395371d52a7"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.719082 4135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:17.719796 4150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.719960 4135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:17.720039 4135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "3624a1a630034194b1f044dc6d2762fa"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "40a99177e5654ac5a8b156c3985c19f8"
server_key_iv: "feeaaa08a674f883716b5395371d52a7"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.720091 4135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:17.729959 4135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:17.731624 4135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:17.731727 4135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:17.731877 4135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:17.732161 4135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:17.732193 4135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.732213 4135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:17.732259 4135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.737651 4135 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:44329
I20260620 19:17:17.737717 4263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:44329 every 8 connection(s)
I20260620 19:17:17.738003 4135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:17.738224 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4135
I20260620 19:17:17.738322 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:17.738536 1031 external_mini_cluster.cc:1428] Setting key 6a83bb5dcf4f60ef829b7ce9b27633d2
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
I20260620 19:17:17.747444 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:17.738788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52467 (local address 127.1.1.254:33943)
0620 19:17:17.738840 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:17.738842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:17.739017 (+ 175us) server_negotiation.cc:407] Connection header received
0620 19:17:17.739588 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:17.739589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:17.739601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:17.739672 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:17.740665 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.741191 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.742240 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.742469 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.742547 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:17.744590 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:17.744591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:17.744592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:17.744600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:17.746056 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.746516 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.746517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.746518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.746581 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.746832 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.746833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.746833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.746946 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:17.747032 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:17.747309 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:17:17.747379 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:17:17.747682 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:17.738653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33943 (local address 127.1.1.193:52467)
0620 19:17:17.738908 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:17.738952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:17.739479 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:17.739719 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:17.739722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:17.740281 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:17.740570 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:17.740576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.741274 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.741275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:17.742149 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:17.742152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.742554 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.742555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:17.742679 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:17.743169 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:17.743177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:17.744497 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:17.746145 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:17.746153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:17.746154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:17.746426 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:17.746658 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:17.746659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:17.746659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:17.746765 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:17.747015 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:17.747021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:17.747180 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 19:17:17.747289 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:17:17.747432 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260620 19:17:17.748225 4264 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33943
I20260620 19:17:17.748323 4264 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:17.748486 4264 heartbeater.cc:511] Master 127.1.1.254:33943 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
I20260620 19:17:17.749025 4073 ts_manager.cc:194] Registered new tserver with Master: 3624a1a630034194b1f044dc6d2762fa (127.1.1.193:44329)
I20260620 19:17:17.749647 4073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:52467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:17.763230 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:17.839553 4269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:17.839761 4269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:17.839795 4269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:17.840947 4269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:17.841046 4269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:17.841115 4269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:17.842509 4269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:17.842758 4269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:17.842962 4269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:17.847769 4269 server_base.cc:1034] running on GCE node
W20260620 19:17:17.847738 4277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.847738 4275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.847708 4274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.848170 4269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:17.848385 4269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:17.849556 4269 hybrid_clock.cc:648] HybridClock initialized: now 1781983037849529 us; error 37 us; skew 500 ppm
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:17.851874 4269 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:17.852262 4269 webserver.cc:460] Webserver started at http://127.1.1.194:36483/ using document root <none> and password file <none>
I20260620 19:17:17.852474 4269 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:17.852535 4269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:17.852674 4269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:17.853596 4269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "6b5e4b8fc4034bc3930fc696a27656e0"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "25d273c22075385535aad7281487740a"
server_key_iv: "a139b4a1289d605d274c1641f0475c1c"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.853909 4269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "6b5e4b8fc4034bc3930fc696a27656e0"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "25d273c22075385535aad7281487740a"
server_key_iv: "a139b4a1289d605d274c1641f0475c1c"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.855062 4269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:17.855700 4284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.855846 4269 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:17.855926 4269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "6b5e4b8fc4034bc3930fc696a27656e0"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "25d273c22075385535aad7281487740a"
server_key_iv: "a139b4a1289d605d274c1641f0475c1c"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.855996 4269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:17.866051 4269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:17.867871 4269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:17.868013 4269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:17.868216 4269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:17.868479 4269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:17.868526 4269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.868561 4269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:17.868611 4269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.873919 4269 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:45179
I20260620 19:17:17.873961 4397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:45179 every 8 connection(s)
I20260620 19:17:17.874199 4269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:17.877242 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4269
I20260620 19:17:17.877333 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:17.877501 1031 external_mini_cluster.cc:1428] Setting key 0ff859e80a5f127f1f80fd023ead5e20
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:17.883037 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:17.874702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33943 (local address 127.1.1.194:52939)
0620 19:17:17.874845 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:17.874879 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:17.875298 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:17.875492 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:17.875495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:17.875976 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:17.876146 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:17.876150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.876742 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.876742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:17.877667 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:17.877669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.878076 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.878077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:17.878195 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:17.878604 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:17.878611 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:17.880037 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:17.881821 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:17.881826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:17.881828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:17.882034 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:17.882245 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:17.882245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:17.882246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:17.882345 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:17.882625 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:17.882628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:17.882722 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:17:17.882795 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:17:17.882895 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260620 19:17:17.883056 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:17.874740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52939 (local address 127.1.1.254:33943)
0620 19:17:17.874799 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:17.874801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:17.874958 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:17:17.875373 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:17.875374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:17.875382 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:17.875434 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:17.876213 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.876672 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.877740 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:17.877980 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:17.878062 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:17.880122 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:17.880124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:17.880124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:17.880132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:17.881744 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.882112 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.882114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.882114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.882177 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:17.882407 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:17.882409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:17.882409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:17.882555 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:17.882634 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:17.882846 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:17:17.882933 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:17:17.883519 4398 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33943
I20260620 19:17:17.883621 4398 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:17.883772 4398 heartbeater.cc:511] Master 127.1.1.254:33943 requested a full tablet report, sending...
I20260620 19:17:17.884127 4073 ts_manager.cc:194] Registered new tserver with Master: 6b5e4b8fc4034bc3930fc696a27656e0 (127.1.1.194:45179)
I20260620 19:17:17.884500 4073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:52939
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:17.900817 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:17.974023 4403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:17.974192 4403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:17.974211 4403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:17.975258 4403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:17.975296 4403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:17.975318 4403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:17.976608 4403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:17.976761 4403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:17.976974 4403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:17.981611 4408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:17.981609 4409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.981740 4403 server_base.cc:1034] running on GCE node
W20260620 19:17:17.981606 4411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:17.981954 4403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:17.982182 4403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:17.983325 4403 hybrid_clock.cc:648] HybridClock initialized: now 1781983037983311 us; error 35 us; skew 500 ppm
Jun 20 19:17:17 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:17.985400 4403 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:17.985842 4403 webserver.cc:460] Webserver started at http://127.1.1.195:36599/ using document root <none> and password file <none>
I20260620 19:17:17.986042 4403 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:17.986102 4403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:17.986229 4403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:17.987062 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "4ef2382e95874cb6973250149070c5d9"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "62d5b02306d52d0460e96c8cab30902a"
server_key_iv: "cc581f5bc8cbb72778b1ff27e4ca1e8a"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.987396 4403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "4ef2382e95874cb6973250149070c5d9"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "62d5b02306d52d0460e96c8cab30902a"
server_key_iv: "cc581f5bc8cbb72778b1ff27e4ca1e8a"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.988559 4403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:17.989219 4418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.989377 4403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:17.989460 4403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "4ef2382e95874cb6973250149070c5d9"
format_stamp: "Formatted at 2026-06-20 19:17:17 on dist-test-slave-mwkb"
server_key: "62d5b02306d52d0460e96c8cab30902a"
server_key_iv: "cc581f5bc8cbb72778b1ff27e4ca1e8a"
server_key_version: "encryptionkey@0"
I20260620 19:17:17.989543 4403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:17.996510 4403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:17.998200 4403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:17.998340 4403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:17.998535 4403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:17.998792 4403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:17.998843 4403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:17.998880 4403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:17.998908 4403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:18.004431 4403 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:33857
I20260620 19:17:18.004477 4531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:33857 every 8 connection(s)
I20260620 19:17:18.004746 4403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983037, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:18.012522 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:18.005405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:44631 (local address 127.1.1.254:33943)
0620 19:17:18.005469 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:18.005471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:18.005602 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:17:18.005941 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:18.005942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:18.005951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:18.005993 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:18.006713 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.007144 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.007960 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.008119 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.008163 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:18.010022 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:18.010023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:18.010024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:18.010031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:18.011430 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:18.011792 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:18.011793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:18.011793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:18.011840 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:18.012037 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:18.012037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:18.012038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:18.012129 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:18.012178 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:18.012390 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:17:18.012445 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:17:18.012655 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:18.005360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33943 (local address 127.1.1.195:44631)
0620 19:17:18.005503 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:18.005535 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:18.005875 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:18.006035 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:18.006038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:18.006490 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:18.006650 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:18.006653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.007211 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.007212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:18.007895 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:18.007897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.008180 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.008181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:18.008261 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:18.008610 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:18.008616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:18.009922 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:18.011519 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:18.011525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:18.011527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:18.011725 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:18.011896 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:18.011897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:18.011897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:18.011983 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:18.012182 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:18.012185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:18.012293 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 19:17:18.012366 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:17:18.012465 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260620 19:17:18.013104 4532 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33943
I20260620 19:17:18.013188 4532 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:18.013306 4532 heartbeater.cc:511] Master 127.1.1.254:33943 requested a full tablet report, sending...
I20260620 19:17:18.013597 4073 ts_manager.cc:194] Registered new tserver with Master: 4ef2382e95874cb6973250149070c5d9 (127.1.1.195:33857)
I20260620 19:17:18.013892 4073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:44631
I20260620 19:17:18.014585 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4403
I20260620 19:17:18.014662 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:18.014781 1031 external_mini_cluster.cc:1428] Setting key 48ff9a092cff072e4ac346a6811aba00
I20260620 19:17:18.015610 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4028](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983038, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:18.036551 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:18.030916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.1.1.254:33943)
0620 19:17:18.030950 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:18.030951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:18.031088 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:17:18.031151 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:18.031152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:18.031159 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:18.031214 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:18.031708 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.032161 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.032774 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:18.032920 (+ 146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:18.032967 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:18.034819 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:18.034821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:18.034821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:18.034826 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:18.035802 (+ 976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:18.036075 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:18.036076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:18.036077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:18.036119 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:18.036218 (+ 99us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:18.036218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:18.036218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:18.036319 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:18.036380 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:18.036451 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:17:18.036498 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:17:18.036747 4073 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42098
I20260620 19:17:18.037206 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4135
I20260620 19:17:18.041955 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4269
I20260620 19:17:18.046128 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4403
I20260620 19:17:18.049948 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4041
2026-06-20T19:17:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1680 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4544](info): set up 2 sockets
Jun 20 19:17:18 dist-test-slave-mwkb krb5kdc[4544](info): commencing operation
krb5kdc: starting...
W20260620 19:17:19.024662 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:19.124645 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35411
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:19.196456 4557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:19.196668 4557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:19.196699 4557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:19.197732 4557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:19.197808 4557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:19.197832 4557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:19.197871 4557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:19.197892 4557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:19.199385 4557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:19.199664 4557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:19.199873 4557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:19.204473 4557 server_base.cc:1034] running on GCE node
W20260620 19:17:19.204429 4565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.204429 4563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.204432 4562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.204902 4557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:19.205224 4557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:19.206373 4557 hybrid_clock.cc:648] HybridClock initialized: now 1781983039206361 us; error 23 us; skew 500 ppm
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:19.208482 4557 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:19.208889 4557 webserver.cc:460] Webserver started at http://127.1.1.254:40223/ using document root <none> and password file <none>
I20260620 19:17:19.209197 4557 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:19.209262 4557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:19.209389 4557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:19.210325 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "1244324f432142c58b01585e5a6b5db0"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "59b802397db863d394d2406ff14935c2"
server_key_iv: "81ddaf43e158bb044f270476c7c53c68"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.210680 4557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "1244324f432142c58b01585e5a6b5db0"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "59b802397db863d394d2406ff14935c2"
server_key_iv: "81ddaf43e158bb044f270476c7c53c68"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.211910 4557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:19.212590 4572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.212755 4557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:19.212846 4557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "1244324f432142c58b01585e5a6b5db0"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "59b802397db863d394d2406ff14935c2"
server_key_iv: "81ddaf43e158bb044f270476c7c53c68"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.212925 4557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:19.222363 4557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:19.223842 4557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:19.223976 4557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:19.227608 4557 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:35411
I20260620 19:17:19.227634 4624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:35411 every 8 connection(s)
I20260620 19:17:19.228071 4557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:19.228349 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4557
I20260620 19:17:19.228467 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:19.228633 1031 external_mini_cluster.cc:1428] Setting key 73922813579249f9bef86a45db631fe8
I20260620 19:17:19.228762 4625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:19.231693 4625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0: Bootstrap starting.
I20260620 19:17:19.232326 4625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:19.232641 4625 log.cc:824] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:19.233322 4625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0: No bootstrap required, opened a new log
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:19.235325 4625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } }
I20260620 19:17:19.235476 4625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:19.235571 4625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1244324f432142c58b01585e5a6b5db0, State: Initialized, Role: FOLLOWER
I20260620 19:17:19.235690 4625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } }
I20260620 19:17:19.235769 4625 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:19.235803 4625 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:19.235841 4625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:19.236390 4625 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } }
I20260620 19:17:19.236491 4625 leader_election.cc:302] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1244324f432142c58b01585e5a6b5db0; no voters:
I20260620 19:17:19.236662 4625 leader_election.cc:288] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:19.236712 4630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:19.236886 4625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:19.236887 4630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [term 1 LEADER]: Becoming Leader. State: Replica: 1244324f432142c58b01585e5a6b5db0, State: Running, Role: LEADER
I20260620 19:17:19.237090 4630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } }
I20260620 19:17:19.237455 4631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1244324f432142c58b01585e5a6b5db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } } }
I20260620 19:17:19.237471 4632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1244324f432142c58b01585e5a6b5db0. Latest consensus state: current_term: 1 leader_uuid: "1244324f432142c58b01585e5a6b5db0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1244324f432142c58b01585e5a6b5db0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35411 } } }
I20260620 19:17:19.237635 4632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:19.237843 4631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:19.237865 4639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:19.238404 4639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:19.239393 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.229086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.1.1.254:35411)
0620 19:17:19.229296 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.229308 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.229344 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:17:19.229773 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.229778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.229825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.229927 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.230674 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.231516 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.232580 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.232958 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.233064 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.235514 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.235523 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.235523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.235535 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.237454 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.237850 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.237853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.237853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.237947 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.238173 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.238173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.238174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.238423 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.239056 (+ 633us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.239116 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.239201 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260620 19:17:19.239867 4639 catalog_manager.cc:1347] Generated new cluster ID: 60a02806839d4bfea34067a0563e0e90
I20260620 19:17:19.239917 4639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:19.254779 4639 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:19.255206 4639 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:19.261157 4639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1244324f432142c58b01585e5a6b5db0: Generated new TSK 0
I20260620 19:17:19.261328 4639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:19.293262 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:19.385370 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:19.385583 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:19.385622 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:19.386667 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:19.386725 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:19.386767 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:19.388165 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:19.388402 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:19.388604 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:19.393002 4656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.392963 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.392982 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.393247 4651 server_base.cc:1034] running on GCE node
I20260620 19:17:19.393441 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:19.393707 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:19.394874 4651 hybrid_clock.cc:648] HybridClock initialized: now 1781983039394863 us; error 22 us; skew 500 ppm
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:19.396988 4651 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:19.397434 4651 webserver.cc:460] Webserver started at http://127.1.1.193:41571/ using document root <none> and password file <none>
I20260620 19:17:19.397679 4651 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:19.397739 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:19.397866 4651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:19.398772 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "aa3d5b39574640d69f62d28e9126319c"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "44d56ac7a4d4fb669641043ff3f262ea"
server_key_iv: "f3b4e8ab1f79652ea5d752c47bec1227"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.399085 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "aa3d5b39574640d69f62d28e9126319c"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "44d56ac7a4d4fb669641043ff3f262ea"
server_key_iv: "f3b4e8ab1f79652ea5d752c47bec1227"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.400377 4651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:19.401132 4666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.401327 4651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:19.401422 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "aa3d5b39574640d69f62d28e9126319c"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "44d56ac7a4d4fb669641043ff3f262ea"
server_key_iv: "f3b4e8ab1f79652ea5d752c47bec1227"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.401525 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:19.414992 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:19.416742 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:19.416889 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:19.417150 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:19.417433 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:19.417496 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.417528 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:19.417555 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.422979 4651 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:36129
I20260620 19:17:19.423013 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:36129 every 8 connection(s)
I20260620 19:17:19.423275 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:19.427775 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4651
I20260620 19:17:19.427879 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:19.428066 1031 external_mini_cluster.cc:1428] Setting key 6eff40ed8efed14cbc6b2e15d9d848c0
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:19.432112 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.423886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60985 (local address 127.1.1.254:35411)
0620 19:17:19.423974 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.423976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.424171 (+ 195us) server_negotiation.cc:407] Connection header received
0620 19:17:19.425015 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.425016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.425027 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.425135 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.425668 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.426133 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.426995 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.427174 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.427219 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.429417 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.429418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.429419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.429426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.430751 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.431127 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.431128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.431128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.431188 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.431452 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.431454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.431454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.431567 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.431642 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.431936 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.432013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 19:17:19.432262 4782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.423880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35411 (local address 127.1.1.193:60985)
0620 19:17:19.424056 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:19.424092 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:19.424889 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:19.425201 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:19.425204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:19.425410 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:19.425592 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.425595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.426206 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.426206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.426930 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.426932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.427237 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.427238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.427322 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.427776 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:19.427783 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:19.429329 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:19.430845 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.430851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.430853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.431044 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.431276 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.431277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.431278 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.431385 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.431635 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:19.431640 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:19.431793 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 19:17:19.431911 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:17:19.432054 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":117,"threads_started":1}
I20260620 19:17:19.432682 4780 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35411
I20260620 19:17:19.432761 4780 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:19.432924 4780 heartbeater.cc:511] Master 127.1.1.254:35411 requested a full tablet report, sending...
I20260620 19:17:19.433348 4589 ts_manager.cc:194] Registered new tserver with Master: aa3d5b39574640d69f62d28e9126319c (127.1.1.193:36129)
I20260620 19:17:19.433882 4589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:60985
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:19.449812 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:19.523995 4785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:19.524175 4785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:19.524195 4785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:19.525318 4785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:19.525357 4785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:19.525383 4785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:19.526877 4785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:19.527067 4785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:19.527287 4785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:19.531972 4790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.531983 4791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.532158 4785 server_base.cc:1034] running on GCE node
W20260620 19:17:19.531988 4793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.532413 4785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:19.532647 4785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:19.533807 4785 hybrid_clock.cc:648] HybridClock initialized: now 1781983039533797 us; error 41 us; skew 500 ppm
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:19.536190 4785 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:19.536581 4785 webserver.cc:460] Webserver started at http://127.1.1.194:43365/ using document root <none> and password file <none>
I20260620 19:17:19.536823 4785 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:19.536890 4785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:19.537057 4785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:19.538160 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "cd97483d149f4ebbb39a8c116ecd1899"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "379319ca061e6a3a25925341072c6909"
server_key_iv: "d8e7feddf82d71b7298c775c83c88964"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.538491 4785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "cd97483d149f4ebbb39a8c116ecd1899"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "379319ca061e6a3a25925341072c6909"
server_key_iv: "d8e7feddf82d71b7298c775c83c88964"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.539718 4785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:19.540371 4800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.540532 4785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:19.540619 4785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "cd97483d149f4ebbb39a8c116ecd1899"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "379319ca061e6a3a25925341072c6909"
server_key_iv: "d8e7feddf82d71b7298c775c83c88964"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.540699 4785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:19.561350 4785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:19.563413 4785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:19.563565 4785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:19.563763 4785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:19.564024 4785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:19.564072 4785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.564107 4785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:19.564132 4785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.569495 4785 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:40119
I20260620 19:17:19.569532 4913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:40119 every 8 connection(s)
I20260620 19:17:19.569841 4785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:19.573791 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4785
I20260620 19:17:19.573900 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:19.574102 1031 external_mini_cluster.cc:1428] Setting key 1db933e02c3440100fb8796b2d064323
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:19.579334 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.570543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35411 (local address 127.1.1.194:59583)
0620 19:17:19.570686 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:19.570716 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:19.571600 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:19.571802 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:19.571805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:19.572040 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:19.572211 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.572215 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.572835 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.572835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.573751 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.573754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.574219 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.574220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.574332 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.574770 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:19.574778 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:19.576235 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:19.577943 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.577948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.577950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.578231 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.578481 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.578482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.578482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.578592 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.578890 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:19.578893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:19.579002 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:17:19.579079 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:17:19.579186 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
I20260620 19:17:19.579360 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.570594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:59583 (local address 127.1.1.254:35411)
0620 19:17:19.570664 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.570666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.570773 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:17:19.571688 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.571689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.571699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.571751 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.572280 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.572757 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.573834 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.574147 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.574214 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.576331 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.576332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.576333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.576341 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.577862 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.578317 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.578318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.578318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.578394 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.578678 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.578679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.578679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.578823 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.578897 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.579132 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.579224 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:17:19.579806 4914 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35411
I20260620 19:17:19.579902 4914 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:19.580089 4914 heartbeater.cc:511] Master 127.1.1.254:35411 requested a full tablet report, sending...
I20260620 19:17:19.580466 4589 ts_manager.cc:194] Registered new tserver with Master: cd97483d149f4ebbb39a8c116ecd1899 (127.1.1.194:40119)
I20260620 19:17:19.580979 4589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:59583
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:19.596663 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:19.669390 4919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:19.669597 4919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:19.669646 4919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:19.670732 4919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:19.670792 4919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:19.670832 4919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:19.672271 4919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:19.672526 4919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:19.672757 4919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:19.677285 4927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:19.677325 4925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.677345 4919 server_base.cc:1034] running on GCE node
W20260620 19:17:19.677285 4924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:19.677704 4919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:19.677940 4919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:19.679106 4919 hybrid_clock.cc:648] HybridClock initialized: now 1781983039679101 us; error 33 us; skew 500 ppm
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:19.681447 4919 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:19.681833 4919 webserver.cc:460] Webserver started at http://127.1.1.195:43755/ using document root <none> and password file <none>
I20260620 19:17:19.682035 4919 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:19.682083 4919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:19.682219 4919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:19.683135 4919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "158cde98638847559e0b1787e35e1241"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "9f1410cafe034547674cb8fda1c297ba"
server_key_iv: "3340323e9fe0168fb4d4ae2dd094b839"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.683449 4919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "158cde98638847559e0b1787e35e1241"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "9f1410cafe034547674cb8fda1c297ba"
server_key_iv: "3340323e9fe0168fb4d4ae2dd094b839"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.684618 4919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:19.685312 4934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.685492 4919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:19.685560 4919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "158cde98638847559e0b1787e35e1241"
format_stamp: "Formatted at 2026-06-20 19:17:19 on dist-test-slave-mwkb"
server_key: "9f1410cafe034547674cb8fda1c297ba"
server_key_iv: "3340323e9fe0168fb4d4ae2dd094b839"
server_key_version: "encryptionkey@0"
I20260620 19:17:19.685640 4919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:19.711355 4919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:19.713451 4919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:19.713593 4919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:19.713786 4919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:19.714079 4919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:19.714110 4919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.714148 4919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:19.714172 4919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:19.719416 4919 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:43731
I20260620 19:17:19.719468 5047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:43731 every 8 connection(s)
I20260620 19:17:19.719789 4919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:19.720369 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 4919
I20260620 19:17:19.720479 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:19.720687 1031 external_mini_cluster.cc:1428] Setting key b53e3ae0d4296f6d4d6692d78be8bd90
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:19.728648 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.720465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:42827 (local address 127.1.1.254:35411)
0620 19:17:19.720517 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.720520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.720760 (+ 240us) server_negotiation.cc:407] Connection header received
0620 19:17:19.721969 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.721970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.721983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.722042 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.722632 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.723111 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.724081 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.724279 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.724348 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.726220 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.726221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.726222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.726229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.727442 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.727835 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.727836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.727836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.727888 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.728097 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.728098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.728098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.728197 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.728264 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.728491 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.728571 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:17:19.728773 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.720417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35411 (local address 127.1.1.195:42827)
0620 19:17:19.720621 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:19.720678 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:19.721864 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:19.722105 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:19.722109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:19.722340 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:19.722544 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.722548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.723199 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.723200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.723969 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:19.723971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.724365 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.724366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:19.724479 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.724890 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:19.724897 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:19.726117 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:19.727558 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.727564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.727566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.727758 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.727944 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:19.727944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:19.727945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:19.728035 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:19.728256 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:19.728262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:19.728370 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 19:17:19.728470 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 19:17:19.728592 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":112,"threads_started":1}
I20260620 19:17:19.729204 5048 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35411
I20260620 19:17:19.729295 5048 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:19.729445 5048 heartbeater.cc:511] Master 127.1.1.254:35411 requested a full tablet report, sending...
I20260620 19:17:19.729786 4589 ts_manager.cc:194] Registered new tserver with Master: 158cde98638847559e0b1787e35e1241 (127.1.1.195:43731)
I20260620 19:17:19.730110 4589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:42827
I20260620 19:17:19.732333 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:19.752722 5055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.746003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.1.1.193:36129)
0620 19:17:19.746165 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.746168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.746174 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:19.746193 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.746194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.746205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.746252 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.746781 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.747446 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.748067 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.748310 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.748355 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.750295 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.750301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.750301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.750307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.751797 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.752094 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.752096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.752096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.752147 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.752305 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.752306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.752306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.752428 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.752525 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.752600 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.752655 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[4544](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:19.759279 4628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:19.753811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.1.1.254:35411)
0620 19:17:19.753855 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:19.753856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:19.753876 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:17:19.753920 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:19.753921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:19.753927 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:19.753964 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:19.754353 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.754821 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.755412 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:19.755595 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:19.755646 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:19.757484 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:19.757485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:19.757485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:19.757491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:19.758519 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.758785 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.758785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.758786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.758827 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:19.758950 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:19.758950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:19.758951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:19.759050 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:19.759108 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:19.759180 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:17:19.759229 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 19:17:19.759474 4589 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59490
I20260620 19:17:19.759855 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4651
I20260620 19:17:19.764106 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4785
I20260620 19:17:19.768296 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4919
I20260620 19:17:19.771970 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 4557
2026-06-20T19:17:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1722 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[5060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[5060](info): set up 2 sockets
Jun 20 19:17:19 dist-test-slave-mwkb krb5kdc[5060](info): commencing operation
krb5kdc: starting...
W20260620 19:17:20.756940 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:20 dist-test-slave-mwkb krb5kdc[5060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:20.860644 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:40775
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:20.932929 5073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:20.933120 5073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:20.933140 5073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:20.934135 5073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:20.934171 5073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:20.934183 5073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:20.934194 5073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:20.934238 5073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:20.935488 5073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:20.935719 5073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:20.935912 5073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:20.940218 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:20.940241 5073 server_base.cc:1034] running on GCE node
W20260620 19:17:20.940209 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:20.940193 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:20.940631 5073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:20.940920 5073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:20.942080 5073 hybrid_clock.cc:648] HybridClock initialized: now 1781983040942051 us; error 40 us; skew 500 ppm
Jun 20 19:17:20 dist-test-slave-mwkb krb5kdc[5060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983040, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:20.944248 5073 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:20.944556 5073 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:20.944623 5073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:20.944787 5073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:20.945787 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "95924782a1e9478b8fd8e91368ed2235"
format_stamp: "Formatted at 2026-06-20 19:17:20 on dist-test-slave-mwkb"
server_key: "aee1b12fdf26f9bc22d5a8ce2a92fb52"
server_key_iv: "ceb97019c50fc7677fd2b1ab445041db"
server_key_version: "encryptionkey@0"
I20260620 19:17:20.946139 5073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "95924782a1e9478b8fd8e91368ed2235"
format_stamp: "Formatted at 2026-06-20 19:17:20 on dist-test-slave-mwkb"
server_key: "aee1b12fdf26f9bc22d5a8ce2a92fb52"
server_key_iv: "ceb97019c50fc7677fd2b1ab445041db"
server_key_version: "encryptionkey@0"
I20260620 19:17:20.947304 5073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:20.947909 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:20.948071 5073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:20.948158 5073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "95924782a1e9478b8fd8e91368ed2235"
format_stamp: "Formatted at 2026-06-20 19:17:20 on dist-test-slave-mwkb"
server_key: "aee1b12fdf26f9bc22d5a8ce2a92fb52"
server_key_iv: "ceb97019c50fc7677fd2b1ab445041db"
server_key_version: "encryptionkey@0"
I20260620 19:17:20.948228 5073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:20.961009 5073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:20.962634 5073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:20.962785 5073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:20.966248 5073 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:40775
I20260620 19:17:20.966281 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:40775 every 8 connection(s)
I20260620 19:17:20.966538 5073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:20.967317 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:20.969354 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: Bootstrap starting.
I20260620 19:17:20.969882 5140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:20.970180 5140 log.cc:824] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:20.970782 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: No bootstrap required, opened a new log
I20260620 19:17:20.972316 5140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } }
I20260620 19:17:20.972440 5140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:20.972498 5140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95924782a1e9478b8fd8e91368ed2235, State: Initialized, Role: FOLLOWER
I20260620 19:17:20.972668 5140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } }
I20260620 19:17:20.972747 5140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:20.972792 5140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:20.972834 5140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:20.973584 5140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } }
I20260620 19:17:20.973682 5140 leader_election.cc:302] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95924782a1e9478b8fd8e91368ed2235; no voters:
I20260620 19:17:20.973821 5140 leader_election.cc:288] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:20.973852 5143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:20.974042 5143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [term 1 LEADER]: Becoming Leader. State: Replica: 95924782a1e9478b8fd8e91368ed2235, State: Running, Role: LEADER
I20260620 19:17:20.974150 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5073
I20260620 19:17:20.974069 5140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:20.974182 5143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } }
I20260620 19:17:20.974236 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:20.974387 1031 external_mini_cluster.cc:1428] Setting key 84cb9b05f50cd39608ff82e400b8d178
I20260620 19:17:20.974538 5144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95924782a1e9478b8fd8e91368ed2235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } } }
I20260620 19:17:20.974543 5145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95924782a1e9478b8fd8e91368ed2235. Latest consensus state: current_term: 1 leader_uuid: "95924782a1e9478b8fd8e91368ed2235" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95924782a1e9478b8fd8e91368ed2235" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40775 } } }
I20260620 19:17:20.974624 5144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:20.974655 5145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:20.975317 5150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:20.975841 5150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:20.977157 5150 catalog_manager.cc:1347] Generated new cluster ID: f0ac4f27fb39498f9f86fa0a884f8b74
I20260620 19:17:20.977203 5150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:17:20 dist-test-slave-mwkb krb5kdc[5060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:20.985803 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:20.974939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58570 (local address 127.1.1.254:40775)
0620 19:17:20.975604 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:20.975612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:20.975644 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:17:20.975766 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:20.975772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:20.975823 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:20.975958 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:20.976767 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:20.978065 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:20.979055 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:20.979727 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:20.979846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:20.982515 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:20.982518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:20.982519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:20.982528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:20.984310 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:20.984706 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:20.984708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:20.984709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:20.984794 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:20.984985 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:20.984986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:20.984986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:20.985258 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:20.985383 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:20.985493 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:17:20.985625 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":83,"threads_started":1}
I20260620 19:17:20.997093 5150 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:20.997707 5150 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:21.008701 5150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: Generated new TSK 0
I20260620 19:17:21.008837 5150 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:21.035924 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:21.109454 5166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.109616 5166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.109637 5166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.110646 5166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.110682 5166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.110729 5166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:21.112142 5166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.112365 5166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.112593 5166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.116822 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.116822 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.116851 5166 server_base.cc:1034] running on GCE node
W20260620 19:17:21.116827 5174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.117228 5166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.117463 5166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.118634 5166 hybrid_clock.cc:648] HybridClock initialized: now 1781983041118624 us; error 22 us; skew 500 ppm
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:21.120946 5166 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:21.121294 5166 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.121380 5166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.121526 5166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.122454 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "171ffed2a53844f39e3d1371b2e944d8"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "2c5282be832d182a81cb010fccb66722"
server_key_iv: "d956d507874cb6d5cb39769a86e79d1d"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.122800 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "171ffed2a53844f39e3d1371b2e944d8"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "2c5282be832d182a81cb010fccb66722"
server_key_iv: "d956d507874cb6d5cb39769a86e79d1d"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.123951 5166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:21.124607 5180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.124789 5166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:21.124873 5166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "171ffed2a53844f39e3d1371b2e944d8"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "2c5282be832d182a81cb010fccb66722"
server_key_iv: "d956d507874cb6d5cb39769a86e79d1d"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.124951 5166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.135725 5166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.137519 5166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.137665 5166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.137842 5166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.138113 5166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.138160 5166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.138196 5166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.138221 5166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.143669 5166 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:42433
I20260620 19:17:21.143697 5293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:42433 every 8 connection(s)
I20260620 19:17:21.143916 5166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:21.149703 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5166
I20260620 19:17:21.149802 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:21.150007 1031 external_mini_cluster.cc:1428] Setting key 0678a894a9073200abe12b25e69c4d08
I20260620 19:17:21.152813 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.144534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52253 (local address 127.1.1.254:40775)
0620 19:17:21.144573 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.144575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.144759 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:17:21.145240 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.145242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.145256 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.145306 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.146089 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.146567 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.147446 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.147642 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.147714 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.149584 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.149586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.149586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:21.149593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.151318 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.151866 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.151867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.151867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.151933 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.152181 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.152182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.152183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.152298 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.152376 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.152641 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.152732 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:21.152935 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.144514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:40775 (local address 127.1.1.193:52253)
0620 19:17:21.144661 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.144689 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.145138 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.145362 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.145365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.145595 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:21.145993 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.145998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.146661 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.146662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.147363 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.147364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.147720 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.147721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.147835 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.148265 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:21.148272 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:21.149499 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:21.151446 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.151452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.151454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.151774 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.151999 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.152000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.152000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.152111 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.152365 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:21.152371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:21.152505 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.152610 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.152743 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 19:17:21.153357 5294 heartbeater.cc:347] Connected to a master server at 127.1.1.254:40775
I20260620 19:17:21.153436 5294 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:21.153589 5294 heartbeater.cc:511] Master 127.1.1.254:40775 requested a full tablet report, sending...
I20260620 19:17:21.154052 5104 ts_manager.cc:194] Registered new tserver with Master: 171ffed2a53844f39e3d1371b2e944d8 (127.1.1.193:42433)
I20260620 19:17:21.154645 5104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:52253
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:21.171772 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:21.246230 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.246382 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.246402 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.247385 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.247421 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.247443 5299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:21.248665 5299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.248811 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.249032 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.253248 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.253310 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.253248 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.253435 5299 server_base.cc:1034] running on GCE node
I20260620 19:17:21.253646 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.253820 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.254949 5299 hybrid_clock.cc:648] HybridClock initialized: now 1781983041254939 us; error 25 us; skew 500 ppm
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:21.257040 5299 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:21.257375 5299 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.257434 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.257560 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.258394 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "a1042b3968894c62bdffebf386c975a5"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "08887d6ae5b42925e6fe3f1ce8858e32"
server_key_iv: "db3f78fe6d872345048130fa1a1518fb"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.258729 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "a1042b3968894c62bdffebf386c975a5"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "08887d6ae5b42925e6fe3f1ce8858e32"
server_key_iv: "db3f78fe6d872345048130fa1a1518fb"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.259864 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:21.260490 5313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.260644 5299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:21.260743 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "a1042b3968894c62bdffebf386c975a5"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "08887d6ae5b42925e6fe3f1ce8858e32"
server_key_iv: "db3f78fe6d872345048130fa1a1518fb"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.260818 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.286432 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.288496 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.288647 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.288836 5299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.289207 5299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.289256 5299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.289317 5299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.289345 5299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.294751 5299 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:45477
I20260620 19:17:21.294782 5426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:45477 every 8 connection(s)
I20260620 19:17:21.294991 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:21.295611 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5299
I20260620 19:17:21.295709 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:21.295871 1031 external_mini_cluster.cc:1428] Setting key 22a25740cf9e030fccd41536c2afa418
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
I20260620 19:17:21.303711 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.295566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:53087 (local address 127.1.1.254:40775)
0620 19:17:21.295603 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.295605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.295826 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:17:21.296301 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.296302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.296311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.296389 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.297329 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.297854 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.298818 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.299039 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.299123 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.301054 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.301056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.301056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:21.301064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.302398 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.302804 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.302805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.302806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.302874 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.303106 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.303107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.303107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.303223 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.303297 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.303546 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.303627 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:21.303866 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.295560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:40775 (local address 127.1.1.194:53087)
0620 19:17:21.295716 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.295754 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.296225 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.296493 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.296497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.296717 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:21.297234 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.297239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.297942 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.297943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.298727 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.298728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.299131 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.299132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.299266 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.299733 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:21.299740 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:21.300964 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:21.302502 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.302508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.302510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.302704 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.302953 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.302953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.302954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.303038 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.303288 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:21.303293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:21.303408 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.303525 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.303657 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260620 19:17:21.304332 5427 heartbeater.cc:347] Connected to a master server at 127.1.1.254:40775
I20260620 19:17:21.304422 5427 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:21.304560 5427 heartbeater.cc:511] Master 127.1.1.254:40775 requested a full tablet report, sending...
I20260620 19:17:21.304908 5104 ts_manager.cc:194] Registered new tserver with Master: a1042b3968894c62bdffebf386c975a5 (127.1.1.194:45477)
I20260620 19:17:21.305315 5104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:53087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:21.317575 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:21.397945 5432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.398108 5432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.398129 5432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.399154 5432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.399191 5432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.399238 5432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:21.400719 5432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.400929 5432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.401168 5432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.405596 5438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.405622 5432 server_base.cc:1034] running on GCE node
W20260620 19:17:21.405596 5440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.405624 5437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.405992 5432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.406263 5432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.407419 5432 hybrid_clock.cc:648] HybridClock initialized: now 1781983041407408 us; error 22 us; skew 500 ppm
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:21.409686 5432 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:21.409982 5432 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.410046 5432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.410235 5432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.411111 5432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "8fb7e786ba434dfbafe68effec40c59e"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "d79702627deaaaf42fa5c7c72aada219"
server_key_iv: "ad564c41e32d5e7abd624342d488f322"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.411448 5432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "8fb7e786ba434dfbafe68effec40c59e"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "d79702627deaaaf42fa5c7c72aada219"
server_key_iv: "ad564c41e32d5e7abd624342d488f322"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.412665 5432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:21.413360 5446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.413512 5432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.413604 5432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "8fb7e786ba434dfbafe68effec40c59e"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "d79702627deaaaf42fa5c7c72aada219"
server_key_iv: "ad564c41e32d5e7abd624342d488f322"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.413676 5432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.433691 5432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.435729 5432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.435871 5432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.436059 5432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.436321 5432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.436369 5432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.436405 5432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.436429 5432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.441799 5432 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:35535
I20260620 19:17:21.441833 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:35535 every 8 connection(s)
I20260620 19:17:21.442068 5432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:17:21 dist-test-slave-mwkb krb5kdc[5060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:21.450084 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.442752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:51577 (local address 127.1.1.254:40775)
0620 19:17:21.442827 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.442829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.442986 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:17:21.443383 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.443384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.443395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.443442 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.444215 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.444674 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.445644 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.445832 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.445909 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.447761 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.447762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.447763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:21.447768 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.448925 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.449310 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.449311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.449312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.449360 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.449548 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.449548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.449548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.449629 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.449689 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.449921 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.449996 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:17:21.450258 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.442684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:40775 (local address 127.1.1.195:51577)
0620 19:17:21.442853 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.442887 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.443290 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.443508 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.443512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.443729 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:21.444124 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.444128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.444777 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.444778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.445552 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.445554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.445917 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.445919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.446025 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.446433 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:21.446439 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:21.447686 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:21.448996 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.449001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.449002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.449251 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.449426 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:21.449426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:21.449427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:21.449510 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:21.449692 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:21.449696 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:21.449799 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.449893 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.450049 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260620 19:17:21.450690 5560 heartbeater.cc:347] Connected to a master server at 127.1.1.254:40775
I20260620 19:17:21.450796 5560 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:21.450925 5560 heartbeater.cc:511] Master 127.1.1.254:40775 requested a full tablet report, sending...
I20260620 19:17:21.451294 5104 ts_manager.cc:194] Registered new tserver with Master: 8fb7e786ba434dfbafe68effec40c59e (127.1.1.195:35535)
I20260620 19:17:21.451637 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5432
I20260620 19:17:21.451668 5104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:51577
I20260620 19:17:21.451737 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:21.451921 1031 external_mini_cluster.cc:1428] Setting key fdbd284857c080de058feded00878833
I20260620 19:17:21.452692 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:21.458833 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.453886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58578 (local address 127.1.1.254:40775)
0620 19:17:21.453920 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.453922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.454097 (+ 175us) server_negotiation.cc:407] Connection header received
0620 19:17:21.454130 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.454131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.454138 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.454170 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.454626 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.455010 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.455610 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.455759 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.455801 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.456773 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.456774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.456775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:21.456780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.458035 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.458305 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.458306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.458306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.458347 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:21.458480 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:21.458481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:21.458481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:21.458577 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.458642 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.458716 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.458764 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:17:21.460139 5104 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:21.461679 5104 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:21.467329 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42433 (local address 127.0.0.1:49518)
0620 19:17:21.463858 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.463868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.463912 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.464075 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.464075 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.464077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:21.464148 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.464150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.465186 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.465188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.466323 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.466326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.467137 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.467138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.467201 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.467210 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.467256 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.467288 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":42,"threads_started":1}
I20260620 19:17:21.467399 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.1.1.193:42433)
0620 19:17:21.463908 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.463912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.463923 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:21.463962 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.463964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.463977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.464240 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:21.464319 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.465034 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.466414 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.467055 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.467119 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.467164 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.467285 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.467322 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260620 19:17:21.467918 5228 tablet_service.cc:1467] Processing CreateTablet for tablet dd131dd263e640bfbe3ae06431d3100b (DEFAULT_TABLE table=test-table [id=8677ee4753c34203a3b3bc647a6786ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:21.468222 5228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd131dd263e640bfbe3ae06431d3100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:21.468282 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:45477 (local address 127.0.0.1:57574)
0620 19:17:21.463637 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.463668 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.463722 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.464002 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.464005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.464007 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:21.464071 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.464072 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.465408 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.465408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.466326 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.466328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.468047 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.468048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.468106 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.468109 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.468173 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.468213 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":99,"threads_started":1}
I20260620 19:17:21.468454 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.1.1.194:45477)
0620 19:17:21.463748 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.463752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.463764 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:17:21.463802 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.463804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.463818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.463894 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:21.464148 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.465313 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.467456 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.467952 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.468011 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.468040 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.468188 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.468225 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260620 19:17:21.469055 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35535 (local address 127.0.0.1:33708)
0620 19:17:21.463978 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.463986 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.464025 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.464210 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.464211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.464212 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:21.464239 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.464239 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.466408 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.466409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.467367 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.467370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.468843 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.468844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.468899 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.468903 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.468956 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.468991 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":61,"threads_started":1}
I20260620 19:17:21.469100 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.463839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.1.1.195:35535)
0620 19:17:21.463968 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.463971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.464018 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:17:21.464076 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.464078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.464089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.464214 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:21.464294 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.465314 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.468092 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.468758 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.468825 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.468868 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.468973 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.469004 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260620 19:17:21.469116 5361 tablet_service.cc:1467] Processing CreateTablet for tablet dd131dd263e640bfbe3ae06431d3100b (DEFAULT_TABLE table=test-table [id=8677ee4753c34203a3b3bc647a6786ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:21.469403 5361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd131dd263e640bfbe3ae06431d3100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:21.469620 5494 tablet_service.cc:1467] Processing CreateTablet for tablet dd131dd263e640bfbe3ae06431d3100b (DEFAULT_TABLE table=test-table [id=8677ee4753c34203a3b3bc647a6786ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:21.469906 5494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd131dd263e640bfbe3ae06431d3100b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:21.471437 5574 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: Bootstrap starting.
I20260620 19:17:21.471762 5575 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Bootstrap starting.
I20260620 19:17:21.472082 5574 tablet_bootstrap.cc:654] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:21.472338 5575 tablet_bootstrap.cc:654] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:21.472649 5574 log.cc:824] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:21.472901 5575 log.cc:824] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:21.473212 5577 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Bootstrap starting.
I20260620 19:17:21.473451 5574 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: No bootstrap required, opened a new log
I20260620 19:17:21.473517 5575 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: No bootstrap required, opened a new log
I20260620 19:17:21.473539 5574 ts_tablet_manager.cc:1397] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:21.473599 5575 ts_tablet_manager.cc:1397] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:21.474967 5574 raft_consensus.cc:348] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.475078 5574 raft_consensus.cc:374] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:21.475097 5574 raft_consensus.cc:729] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171ffed2a53844f39e3d1371b2e944d8, State: Initialized, Role: FOLLOWER
I20260620 19:17:21.475059 5575 raft_consensus.cc:348] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.475207 5574 consensus_queue.cc:260] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.475242 5575 raft_consensus.cc:374] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:21.475278 5575 raft_consensus.cc:729] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1042b3968894c62bdffebf386c975a5, State: Initialized, Role: FOLLOWER
I20260620 19:17:21.475426 5575 consensus_queue.cc:260] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.475468 5574 ts_tablet_manager.cc:1428] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:21.475488 5294 heartbeater.cc:503] Master 127.1.1.254:40775 was elected leader, sending a full tablet report...
I20260620 19:17:21.475695 5575 ts_tablet_manager.cc:1428] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:17:21.475698 5427 heartbeater.cc:503] Master 127.1.1.254:40775 was elected leader, sending a full tablet report...
I20260620 19:17:21.476240 5577 tablet_bootstrap.cc:654] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:21.476780 5577 log.cc:824] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:21.477424 5577 tablet_bootstrap.cc:492] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: No bootstrap required, opened a new log
I20260620 19:17:21.477495 5577 ts_tablet_manager.cc:1397] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:17:21.478650 5577 raft_consensus.cc:348] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.478778 5577 raft_consensus.cc:374] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:21.478820 5577 raft_consensus.cc:729] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb7e786ba434dfbafe68effec40c59e, State: Initialized, Role: FOLLOWER
I20260620 19:17:21.478924 5577 consensus_queue.cc:260] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.479182 5560 heartbeater.cc:503] Master 127.1.1.254:40775 was elected leader, sending a full tablet report...
I20260620 19:17:21.479187 5577 ts_tablet_manager.cc:1428] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:17:21.487617 5580 raft_consensus.cc:482] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:21.487701 5580 raft_consensus.cc:504] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.487977 5580 leader_election.cc:288] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 171ffed2a53844f39e3d1371b2e944d8 (127.1.1.193:42433), 8fb7e786ba434dfbafe68effec40c59e (127.1.1.195:35535)
I20260620 19:17:21.491134 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.488192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:51417 (local address 127.1.1.195:35535)
0620 19:17:21.488220 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.488222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.488349 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:17:21.488394 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.488395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.488402 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.488450 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:21.488554 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.489171 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490346 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.490885 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490938 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.490944 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.491055 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.491083 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:21.491134 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.488123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:49143 (local address 127.1.1.193:42433)
0620 19:17:21.488160 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.488162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.488170 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:21.488224 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.488225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.488234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.488284 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:21.488398 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.489023 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490292 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.490831 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490896 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.490904 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.491055 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.491084 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:17:21.491142 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.488086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42433 (local address 127.1.1.194:49143)
0620 19:17:21.488126 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.488136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.488173 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.488302 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.488303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.488305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:21.488341 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.488342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.489115 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.489115 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.490216 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.490218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.490898 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.490968 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.490971 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.491033 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.491074 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":8}
I20260620 19:17:21.491146 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.488127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35535 (local address 127.1.1.194:51417)
0620 19:17:21.488284 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.488293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.488336 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.488460 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.488460 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.488461 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:21.488498 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.488499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.489240 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.489241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.490281 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:21.490283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.490943 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.490943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:21.490997 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.490999 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.491046 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.491074 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260620 19:17:21.491468 5248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd131dd263e640bfbe3ae06431d3100b" candidate_uuid: "a1042b3968894c62bdffebf386c975a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171ffed2a53844f39e3d1371b2e944d8" is_pre_election: true
I20260620 19:17:21.491600 5248 raft_consensus.cc:2454] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1042b3968894c62bdffebf386c975a5 in term 0.
I20260620 19:17:21.491714 5514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd131dd263e640bfbe3ae06431d3100b" candidate_uuid: "a1042b3968894c62bdffebf386c975a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb7e786ba434dfbafe68effec40c59e" is_pre_election: true
I20260620 19:17:21.491763 5315 leader_election.cc:302] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171ffed2a53844f39e3d1371b2e944d8, a1042b3968894c62bdffebf386c975a5; no voters:
I20260620 19:17:21.491799 5514 raft_consensus.cc:2454] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1042b3968894c62bdffebf386c975a5 in term 0.
I20260620 19:17:21.491868 5580 raft_consensus.cc:2781] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:21.491925 5580 raft_consensus.cc:482] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:21.491953 5580 raft_consensus.cc:3037] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:21.492614 5580 raft_consensus.cc:504] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.492740 5580 leader_election.cc:288] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [CANDIDATE]: Term 1 election: Requested vote from peers 171ffed2a53844f39e3d1371b2e944d8 (127.1.1.193:42433), 8fb7e786ba434dfbafe68effec40c59e (127.1.1.195:35535)
I20260620 19:17:21.492862 5514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd131dd263e640bfbe3ae06431d3100b" candidate_uuid: "a1042b3968894c62bdffebf386c975a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fb7e786ba434dfbafe68effec40c59e"
I20260620 19:17:21.492851 5248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd131dd263e640bfbe3ae06431d3100b" candidate_uuid: "a1042b3968894c62bdffebf386c975a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171ffed2a53844f39e3d1371b2e944d8"
I20260620 19:17:21.492918 5514 raft_consensus.cc:3037] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:21.492918 5248 raft_consensus.cc:3037] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:21.493618 5248 raft_consensus.cc:2454] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1042b3968894c62bdffebf386c975a5 in term 1.
I20260620 19:17:21.493618 5514 raft_consensus.cc:2454] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1042b3968894c62bdffebf386c975a5 in term 1.
I20260620 19:17:21.493778 5315 leader_election.cc:302] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171ffed2a53844f39e3d1371b2e944d8, a1042b3968894c62bdffebf386c975a5; no voters:
I20260620 19:17:21.493872 5580 raft_consensus.cc:2781] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:21.493983 5580 raft_consensus.cc:686] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 1 LEADER]: Becoming Leader. State: Replica: a1042b3968894c62bdffebf386c975a5, State: Running, Role: LEADER
I20260620 19:17:21.494062 5580 consensus_queue.cc:237] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } }
I20260620 19:17:21.494669 5104 catalog_manager.cc:5495] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1042b3968894c62bdffebf386c975a5 (127.1.1.194). New cstate: current_term: 1 leader_uuid: "a1042b3968894c62bdffebf386c975a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1042b3968894c62bdffebf386c975a5" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 45477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:21.503091 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.501235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.1.1.194:45477)
0620 19:17:21.501285 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.501287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.501293 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:21.501339 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.501340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.501352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.501417 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:21.501500 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.501897 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.502541 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:21.502716 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:21.502767 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:21.502871 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:21.502941 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:21.502997 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.503013 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.503026 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:17:21.504333 5248 raft_consensus.cc:1261] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 1 FOLLOWER]: Refusing update from remote peer a1042b3968894c62bdffebf386c975a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:21.504347 5514 raft_consensus.cc:1261] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 1 FOLLOWER]: Refusing update from remote peer a1042b3968894c62bdffebf386c975a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:21.504601 5580 consensus_queue.cc:1035] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb7e786ba434dfbafe68effec40c59e" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:21.504699 5584 consensus_queue.cc:1035] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "171ffed2a53844f39e3d1371b2e944d8" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:21.506403 5587 mvcc.cc:205] Tried to move back new op lower bound from 7299002537999962112 to 7299002537960177664. Current Snapshot: MvccSnapshot[applied={T|T < 7299002537999962112}]
I20260620 19:17:21.507143 5588 mvcc.cc:205] Tried to move back new op lower bound from 7299002537999962112 to 7299002537960177664. Current Snapshot: MvccSnapshot[applied={T|T < 7299002537999962112 or (T in {7299002537999962112})}]
I20260620 19:17:21.507423 5589 mvcc.cc:205] Tried to move back new op lower bound from 7299002537999962112 to 7299002537960177664. Current Snapshot: MvccSnapshot[applied={T|T < 7299002537999962112}]
I20260620 19:17:21.508993 5102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:21.509107 5102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:21.509701 5102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95924782a1e9478b8fd8e91368ed2235: Sending DeleteTablet for 3 replicas of tablet dd131dd263e640bfbe3ae06431d3100b
I20260620 19:17:21.509965 5361 tablet_service.cc:1514] Processing DeleteTablet for tablet dd131dd263e640bfbe3ae06431d3100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:21 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:57574
I20260620 19:17:21.510018 5228 tablet_service.cc:1514] Processing DeleteTablet for tablet dd131dd263e640bfbe3ae06431d3100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:21 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:49518
I20260620 19:17:21.510016 5494 tablet_service.cc:1514] Processing DeleteTablet for tablet dd131dd263e640bfbe3ae06431d3100b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:21 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:33708
I20260620 19:17:21.510217 5597 tablet_replica.cc:330] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8: stopping tablet replica
I20260620 19:17:21.510304 5597 raft_consensus.cc:2229] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:21.510350 5596 tablet_replica.cc:330] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: stopping tablet replica
I20260620 19:17:21.510381 5597 raft_consensus.cc:2258] T dd131dd263e640bfbe3ae06431d3100b P 171ffed2a53844f39e3d1371b2e944d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:21.510407 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5166
I20260620 19:17:21.510432 5596 raft_consensus.cc:2229] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:21.511126 5596 raft_consensus.cc:2258] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:21.511720 5596 ts_tablet_manager.cc:1905] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:21.512140 5598 tablet_replica.cc:330] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: stopping tablet replica
I20260620 19:17:21.512220 5598 raft_consensus.cc:2229] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:21.512300 5598 raft_consensus.cc:2258] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:21.512712 5598 ts_tablet_manager.cc:1905] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:21.513437 5596 ts_tablet_manager.cc:1918] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:21.513509 5596 log.cc:1196] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/dd131dd263e640bfbe3ae06431d3100b
I20260620 19:17:21.513758 5596 ts_tablet_manager.cc:1939] T dd131dd263e640bfbe3ae06431d3100b P a1042b3968894c62bdffebf386c975a5: Deleting consensus metadata
I20260620 19:17:21.513981 5598 ts_tablet_manager.cc:1918] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:21.514058 5598 log.cc:1196] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/dd131dd263e640bfbe3ae06431d3100b
I20260620 19:17:21.514194 5090 catalog_manager.cc:4841] TS a1042b3968894c62bdffebf386c975a5 (127.1.1.194:45477): tablet dd131dd263e640bfbe3ae06431d3100b (table test-table [id=8677ee4753c34203a3b3bc647a6786ef]) successfully deleted
I20260620 19:17:21.514433 5598 ts_tablet_manager.cc:1939] T dd131dd263e640bfbe3ae06431d3100b P 8fb7e786ba434dfbafe68effec40c59e: Deleting consensus metadata
I20260620 19:17:21.514784 5090 catalog_manager.cc:4841] TS 8fb7e786ba434dfbafe68effec40c59e (127.1.1.195:35535): tablet dd131dd263e640bfbe3ae06431d3100b (table test-table [id=8677ee4753c34203a3b3bc647a6786ef]) successfully deleted
W20260620 19:17:21.516618 5089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:42433 (error 108)
I20260620 19:17:21.517015 5569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.516921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42433 (local address 127.0.0.1:49524)
0620 19:17:21.516960 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.516990 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:17:21.517119 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5299
W20260620 19:17:21.517171 5089 catalog_manager.cc:4584] TS 171ffed2a53844f39e3d1371b2e944d8 (127.1.1.193:42433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd131dd263e640bfbe3ae06431d3100b: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42433: connect: Connection refused (error 111)
I20260620 19:17:21.521620 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5432
I20260620 19:17:21.525610 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5073
2026-06-20T19:17:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1753 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:17:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:21Z Disabled control of system clock
I20260620 19:17:21.551471 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:45513
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:45513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:17:21.624433 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.624627 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.624657 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.625701 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:21.625761 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.625780 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.625798 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:21.625825 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:21.626963 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:45513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:45513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.627219 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.627470 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:21.631804 5603 server_base.cc:1034] running on GCE node
W20260620 19:17:21.631776 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.631780 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.631752 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.632193 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.632460 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.633617 5603 hybrid_clock.cc:648] HybridClock initialized: now 1781983041633614 us; error 45 us; skew 500 ppm
I20260620 19:17:21.634616 5603 webserver.cc:460] Webserver started at http://127.1.1.254:39663/ using document root <none> and password file <none>
I20260620 19:17:21.634824 5603 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.634881 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.634991 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.635859 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "2ba2f75b78d84d01a7092b0941613040"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "f0eab4622cc37c5056b6355fc87d599c"
server_key_iv: "a302ffb017b05439f2599f2c7eae6066"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.636171 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "2ba2f75b78d84d01a7092b0941613040"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "f0eab4622cc37c5056b6355fc87d599c"
server_key_iv: "a302ffb017b05439f2599f2c7eae6066"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.637341 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:21.637982 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.638139 5603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.638226 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2ba2f75b78d84d01a7092b0941613040"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "f0eab4622cc37c5056b6355fc87d599c"
server_key_iv: "a302ffb017b05439f2599f2c7eae6066"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.638298 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.641263 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.641489 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.641640 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.645002 5603 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:45513
I20260620 19:17:21.645028 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:45513 every 8 connection(s)
I20260620 19:17:21.645351 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:21.645879 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:21.648058 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: Bootstrap starting.
I20260620 19:17:21.648492 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:21.648700 5670 log.cc:824] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:21.649312 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: No bootstrap required, opened a new log
I20260620 19:17:21.650599 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } }
I20260620 19:17:21.650710 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:21.650730 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba2f75b78d84d01a7092b0941613040, State: Initialized, Role: FOLLOWER
I20260620 19:17:21.650841 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } }
I20260620 19:17:21.650914 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:21.650934 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:21.650964 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:21.651396 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } }
I20260620 19:17:21.651482 5670 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba2f75b78d84d01a7092b0941613040; no voters:
I20260620 19:17:21.651594 5670 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:21.651669 5673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:21.651805 5673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba2f75b78d84d01a7092b0941613040, State: Running, Role: LEADER
I20260620 19:17:21.651824 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:21.651887 5673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } }
I20260620 19:17:21.652275 5675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba2f75b78d84d01a7092b0941613040. Latest consensus state: current_term: 1 leader_uuid: "2ba2f75b78d84d01a7092b0941613040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } } }
I20260620 19:17:21.652360 5675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:21.652635 5674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba2f75b78d84d01a7092b0941613040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba2f75b78d84d01a7092b0941613040" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 45513 } } }
I20260620 19:17:21.652755 5674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:21.652899 5682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:21.653326 5682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:21.654759 5682 catalog_manager.cc:1347] Generated new cluster ID: 5ffc3a9e3d1e4937848ad45643c924a4
I20260620 19:17:21.654810 5682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:21.655045 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5603
I20260620 19:17:21.655141 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:21.655303 1031 external_mini_cluster.cc:1428] Setting key dac09e4806e9567a7c9c1f75e25773b6
I20260620 19:17:21.658494 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.655710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.1.1.254:45513)
0620 19:17:21.657205 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.657212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.657234 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:17:21.657308 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.657312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.657339 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.657464 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.658119 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.658121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.658122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:21.658124 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.658155 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:21.658163 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.658229 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.658292 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.658347 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":63,"threads_started":1}
I20260620 19:17:21.666092 5682 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:21.666236 5682 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:21.673197 5682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: Generated new TSK 0
I20260620 19:17:21.673333 5682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:17:21.676087 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--builtin_ntp_servers=127.1.1.212:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:17:21.761830 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.762012 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.762034 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.763154 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.763190 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.763214 5694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:21.764525 5694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.764773 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.765100 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.770131 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.770147 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.770133 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.770274 5694 server_base.cc:1034] running on GCE node
I20260620 19:17:21.770489 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.770718 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.771878 5694 hybrid_clock.cc:648] HybridClock initialized: now 1781983041771855 us; error 38 us; skew 500 ppm
I20260620 19:17:21.772976 5694 webserver.cc:460] Webserver started at http://127.1.1.193:44121/ using document root <none> and password file <none>
I20260620 19:17:21.773203 5694 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.773262 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.773406 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.774267 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "cbdc9babef8842bc8d6d2d81db078e1b"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "05012d8112943026b8b52f8f40c9a70e"
server_key_iv: "ad464c6cb96906d466083db78e398e9a"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.774585 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "cbdc9babef8842bc8d6d2d81db078e1b"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "05012d8112943026b8b52f8f40c9a70e"
server_key_iv: "ad464c6cb96906d466083db78e398e9a"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.775741 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:21.776396 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.776551 5694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:21.776633 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "cbdc9babef8842bc8d6d2d81db078e1b"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "05012d8112943026b8b52f8f40c9a70e"
server_key_iv: "ad464c6cb96906d466083db78e398e9a"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.776706 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.779661 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.779909 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.780050 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.780254 5694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.780582 5694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.780633 5694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.780671 5694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.780699 5694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.786105 5694 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:40909
I20260620 19:17:21.786139 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:40909 every 8 connection(s)
I20260620 19:17:21.786371 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:21.788977 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.786941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:38283 (local address 127.1.1.254:45513)
0620 19:17:21.786980 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.786982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.787175 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:17:21.787638 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.787639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.787644 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.787687 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.788802 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.788803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.788803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:21.788807 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.788818 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:21.788822 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.788857 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.788924 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.788935 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:21.789261 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.786923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:45513 (local address 127.1.1.193:38283)
0620 19:17:21.787064 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.787097 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.787562 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.787747 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.787751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.787759 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:21.788639 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:21.788639 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:21.788675 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:21.788709 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:21.788860 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:21.788864 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:21.788867 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.788934 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.788985 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260620 19:17:21.789690 5822 heartbeater.cc:347] Connected to a master server at 127.1.1.254:45513
I20260620 19:17:21.789783 5822 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:21.789897 5822 heartbeater.cc:511] Master 127.1.1.254:45513 requested a full tablet report, sending...
I20260620 19:17:21.790304 5634 ts_manager.cc:194] Registered new tserver with Master: cbdc9babef8842bc8d6d2d81db078e1b (127.1.1.193:40909)
I20260620 19:17:21.790743 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5694
I20260620 19:17:21.790832 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:21.791000 1031 external_mini_cluster.cc:1428] Setting key 2f2b07ab38be1a0c929f05a56ae38d24
I20260620 19:17:21.791882 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--builtin_ntp_servers=127.1.1.212:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:17:21.863966 5825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.864126 5825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.864146 5825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.865183 5825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.865216 5825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.865237 5825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:21.866365 5825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.866568 5825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.866812 5825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.871181 5830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.871184 5831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.871191 5833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.871366 5825 server_base.cc:1034] running on GCE node
I20260620 19:17:21.871610 5825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.871836 5825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.872991 5825 hybrid_clock.cc:648] HybridClock initialized: now 1781983041872980 us; error 24 us; skew 500 ppm
I20260620 19:17:21.874117 5825 webserver.cc:460] Webserver started at http://127.1.1.194:41141/ using document root <none> and password file <none>
I20260620 19:17:21.874318 5825 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.874378 5825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.874502 5825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.875450 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "320f089fa05f42f6be1232d14120fbad"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "ebee1a3ba5824e96f520577971149fe9"
server_key_iv: "2d65b52d7f5b9c907057576adb9b8998"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.875780 5825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "320f089fa05f42f6be1232d14120fbad"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "ebee1a3ba5824e96f520577971149fe9"
server_key_iv: "2d65b52d7f5b9c907057576adb9b8998"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.876964 5825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:21.877604 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.877749 5825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.877832 5825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "320f089fa05f42f6be1232d14120fbad"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "ebee1a3ba5824e96f520577971149fe9"
server_key_iv: "2d65b52d7f5b9c907057576adb9b8998"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.877908 5825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.880911 5825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.881202 5825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.881336 5825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.881562 5825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.881839 5825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.881891 5825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.881935 5825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.881963 5825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.887491 5825 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:41145
I20260620 19:17:21.887542 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:41145 every 8 connection(s)
I20260620 19:17:21.887840 5825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:21.890419 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.888500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52015 (local address 127.1.1.254:45513)
0620 19:17:21.888562 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:21.888566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:21.888779 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:17:21.889238 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:21.889239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:21.889243 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:21.889297 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:21.890246 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:21.890247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:21.890247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:21.890251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:21.890265 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:21.890269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:21.890311 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:21.890358 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:17:21.890370 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:17:21.890622 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:21.888497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:45513 (local address 127.1.1.194:52015)
0620 19:17:21.888672 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:21.888705 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:21.889161 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:21.889341 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:21.889344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:21.889352 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:21.890127 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:21.890127 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:21.890151 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:21.890174 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:21.890314 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:21.890316 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:21.890318 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:21.890364 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:17:21.890405 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260620 19:17:21.891084 5953 heartbeater.cc:347] Connected to a master server at 127.1.1.254:45513
I20260620 19:17:21.891165 5953 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:21.891260 5953 heartbeater.cc:511] Master 127.1.1.254:45513 requested a full tablet report, sending...
I20260620 19:17:21.891566 5634 ts_manager.cc:194] Registered new tserver with Master: 320f089fa05f42f6be1232d14120fbad (127.1.1.194:41145)
I20260620 19:17:21.895889 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5825
I20260620 19:17:21.895965 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:21.896078 1031 external_mini_cluster.cc:1428] Setting key c1c430118fa864bcdf0a7d535b3eb5c3
I20260620 19:17:21.896806 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--builtin_ntp_servers=127.1.1.212:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:17:21.976275 5956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:21.976460 5956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:21.976482 5956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:21.977612 5956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:21.977654 5956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:21.977680 5956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:21.979004 5956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:45513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:21.979204 5956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:21.979444 5956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:21.984261 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:21.984266 5962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.984303 5956 server_base.cc:1034] running on GCE node
W20260620 19:17:21.984269 5961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:21.984599 5956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:21.984875 5956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:21.986027 5956 hybrid_clock.cc:648] HybridClock initialized: now 1781983041986024 us; error 34 us; skew 500 ppm
I20260620 19:17:21.987164 5956 webserver.cc:460] Webserver started at http://127.1.1.195:43045/ using document root <none> and password file <none>
I20260620 19:17:21.987409 5956 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:21.987489 5956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:21.987622 5956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:21.988582 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "9ce6193df1fd40e7841aa615ad204db6"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "062265a2dad7c94e556a9afced51502f"
server_key_iv: "93e273b92c2693df3a66ede63f2f23e1"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.988934 5956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "9ce6193df1fd40e7841aa615ad204db6"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "062265a2dad7c94e556a9afced51502f"
server_key_iv: "93e273b92c2693df3a66ede63f2f23e1"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.990200 5956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:21.990942 5970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.991107 5956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:21.991199 5956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "9ce6193df1fd40e7841aa615ad204db6"
format_stamp: "Formatted at 2026-06-20 19:17:21 on dist-test-slave-mwkb"
server_key: "062265a2dad7c94e556a9afced51502f"
server_key_iv: "93e273b92c2693df3a66ede63f2f23e1"
server_key_version: "encryptionkey@0"
I20260620 19:17:21.991284 5956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:21.994464 5956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:21.994714 5956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:21.994844 5956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:21.995050 5956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:21.995326 5956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:21.995380 5956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:21.995419 5956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:21.995446 5956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.000852 5956 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:36775
I20260620 19:17:22.000901 6083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:36775 every 8 connection(s)
I20260620 19:17:22.001252 5956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:22.003744 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.001919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:58701 (local address 127.1.1.254:45513)
0620 19:17:22.001984 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.001986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.002199 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:17:22.002573 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.002574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.002578 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.002628 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.003569 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.003570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.003571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.003574 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.003589 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.003593 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.003640 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.003684 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.003696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:17:22.003935 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.001911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:45513 (local address 127.1.1.195:58701)
0620 19:17:22.002085 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.002120 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.002498 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.002673 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.002676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.002683 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.003438 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.003439 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.003462 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.003491 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.003636 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.003638 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.003640 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.003682 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.003722 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260620 19:17:22.004451 6084 heartbeater.cc:347] Connected to a master server at 127.1.1.254:45513
I20260620 19:17:22.004549 6084 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:22.004681 6084 heartbeater.cc:511] Master 127.1.1.254:45513 requested a full tablet report, sending...
I20260620 19:17:22.005038 5634 ts_manager.cc:194] Registered new tserver with Master: 9ce6193df1fd40e7841aa615ad204db6 (127.1.1.195:36775)
I20260620 19:17:22.011006 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 5956
I20260620 19:17:22.011089 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:22.011270 1031 external_mini_cluster.cc:1428] Setting key 2c084f88f0fde3647f40b0d6c77b7a05
I20260620 19:17:22.012108 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:22.014272 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.013382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.1.1.254:45513)
0620 19:17:22.013422 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.013424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.013547 (+ 123us) server_negotiation.cc:407] Connection header received
0620 19:17:22.013569 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.013570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.013574 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.013626 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.014062 (+ 436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.014063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.014064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.014067 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.014080 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.014083 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.014123 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.014158 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.014169 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:22.015655 5634 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:22.017004 5634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:22.020485 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.019183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:41145 (local address 127.0.0.1:33584)
0620 19:17:22.019440 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.019449 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.019479 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.019619 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.019620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.019623 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.020096 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.020096 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.020111 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.020131 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.020341 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.020346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.020349 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.020412 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.020425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":61,"threads_started":1}
I20260620 19:17:22.020696 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.018982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36775 (local address 127.0.0.1:38742)
0620 19:17:22.019298 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.019306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.019341 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.019929 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.019930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.019932 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.020301 (+ 369us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.020301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.020325 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.020336 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.020536 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.020537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.020538 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.020592 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.020604 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":95,"threads_started":1}
I20260620 19:17:22.020771 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.018982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40909 (local address 127.0.0.1:56562)
0620 19:17:22.019140 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.019171 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.019219 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.019472 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.019479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.019484 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.020330 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.020330 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.020338 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.020352 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.020589 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.020591 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.020591 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.020624 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.020636 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260620 19:17:22.020777 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.019269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33584 (local address 127.1.1.194:41145)
0620 19:17:22.019469 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.019473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.019484 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:22.019535 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.019538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.019547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.019609 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.020198 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.020205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.020205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.020209 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.020241 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.020262 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.020338 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.020662 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.020680 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260620 19:17:22.020869 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.019134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.1.1.193:40909)
0620 19:17:22.019311 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.019316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.019327 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:17:22.019355 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.019359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.019368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.019458 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.020432 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.020439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.020439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.020443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.020487 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.020508 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.020736 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.020758 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.020773 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260620 19:17:22.020994 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.019047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.1.1.195:36775)
0620 19:17:22.019808 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.019812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.019820 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:22.019841 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.019843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.019851 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.019925 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.020393 (+ 468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.020398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.020399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.020403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.020433 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.020453 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.020520 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.020907 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.020922 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":71,"threads_started":1}
I20260620 19:17:22.021438 5887 tablet_service.cc:1467] Processing CreateTablet for tablet 37e75505a23143de898fbc3870f8e168 (DEFAULT_TABLE table=test-table [id=2621b6aaf5374c2592b120234074bd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:22.021579 5756 tablet_service.cc:1467] Processing CreateTablet for tablet 37e75505a23143de898fbc3870f8e168 (DEFAULT_TABLE table=test-table [id=2621b6aaf5374c2592b120234074bd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:22.021788 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e75505a23143de898fbc3870f8e168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.021867 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e75505a23143de898fbc3870f8e168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.024258 6099 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Bootstrap starting.
I20260620 19:17:22.024602 6018 tablet_service.cc:1467] Processing CreateTablet for tablet 37e75505a23143de898fbc3870f8e168 (DEFAULT_TABLE table=test-table [id=2621b6aaf5374c2592b120234074bd67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:22.024878 6018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e75505a23143de898fbc3870f8e168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.024883 6099 tablet_bootstrap.cc:654] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.025229 6099 log.cc:824] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.025550 6098 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Bootstrap starting.
I20260620 19:17:22.025990 6099 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: No bootstrap required, opened a new log
I20260620 19:17:22.026067 6099 ts_tablet_manager.cc:1397] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:17:22.026086 6098 tablet_bootstrap.cc:654] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.026342 6098 log.cc:824] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.027001 6098 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: No bootstrap required, opened a new log
I20260620 19:17:22.027063 6098 ts_tablet_manager.cc:1397] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:22.027838 6102 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Bootstrap starting.
I20260620 19:17:22.028476 6102 tablet_bootstrap.cc:654] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.028683 6099 raft_consensus.cc:348] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.028779 6102 log.cc:824] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.028823 6099 raft_consensus.cc:374] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.028864 6099 raft_consensus.cc:729] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320f089fa05f42f6be1232d14120fbad, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.028975 6099 consensus_queue.cc:260] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.029263 6099 ts_tablet_manager.cc:1428] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:17:22.029403 5953 heartbeater.cc:503] Master 127.1.1.254:45513 was elected leader, sending a full tablet report...
I20260620 19:17:22.029524 6102 tablet_bootstrap.cc:492] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: No bootstrap required, opened a new log
I20260620 19:17:22.029584 6102 ts_tablet_manager.cc:1397] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:22.029963 6098 raft_consensus.cc:348] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.030126 6098 raft_consensus.cc:374] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.030166 6098 raft_consensus.cc:729] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbdc9babef8842bc8d6d2d81db078e1b, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.030273 6098 consensus_queue.cc:260] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.030499 6098 ts_tablet_manager.cc:1428] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:17:22.030536 5822 heartbeater.cc:503] Master 127.1.1.254:45513 was elected leader, sending a full tablet report...
I20260620 19:17:22.031981 6102 raft_consensus.cc:348] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.032105 6102 raft_consensus.cc:374] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.032126 6102 raft_consensus.cc:729] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ce6193df1fd40e7841aa615ad204db6, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.032248 6102 consensus_queue.cc:260] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.032487 6102 ts_tablet_manager.cc:1428] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:17:22.032501 6084 heartbeater.cc:503] Master 127.1.1.254:45513 was elected leader, sending a full tablet report...
W20260620 19:17:22.036859 5823 tablet.cc:2369] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:17:22.138331 5954 tablet.cc:2369] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:17:22.203639 6104 raft_consensus.cc:482] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:22.203737 6104 raft_consensus.cc:504] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.204005 6104 leader_election.cc:288] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ce6193df1fd40e7841aa615ad204db6 (127.1.1.195:36775), cbdc9babef8842bc8d6d2d81db078e1b (127.1.1.193:40909)
I20260620 19:17:22.205221 6094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.204251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:41173 (local address 127.1.1.195:36775)
0620 19:17:22.204292 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.204294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.204303 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:17:22.204332 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.204333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.204339 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.204416 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.204911 (+ 495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.204912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.204912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.204916 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.204954 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.204957 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.205034 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.205153 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.205167 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:22.205219 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.204188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:36775 (local address 127.1.1.194:41173)
0620 19:17:22.204247 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.204258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.204291 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.204431 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.204432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.204434 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.204797 (+ 363us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.204797 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.204804 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.204814 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.205041 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.205041 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.205042 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.205140 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.205154 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:17:22.205277 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.204262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:36445 (local address 127.1.1.193:40909)
0620 19:17:22.204299 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.204301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.204385 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:17:22.204462 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.204463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.204467 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.204512 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.205021 (+ 509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.205023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.205023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.205028 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.205044 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.205048 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.205142 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.205201 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.205212 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:22.205338 6107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.204188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40909 (local address 127.1.1.194:36445)
0620 19:17:22.204326 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.204334 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.204374 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.204532 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.204533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.204534 (+ 1us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:22.204910 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:17:22.204910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:22.204914 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:17:22.204939 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:22.205144 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:22.205146 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:22.205146 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:17:22.205192 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:17:22.205205 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":11,"thread_start_us":75,"threads_started":1}
I20260620 19:17:22.205533 6038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e75505a23143de898fbc3870f8e168" candidate_uuid: "320f089fa05f42f6be1232d14120fbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce6193df1fd40e7841aa615ad204db6" is_pre_election: true
I20260620 19:17:22.205684 6038 raft_consensus.cc:2454] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 320f089fa05f42f6be1232d14120fbad in term 0.
I20260620 19:17:22.205648 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e75505a23143de898fbc3870f8e168" candidate_uuid: "320f089fa05f42f6be1232d14120fbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" is_pre_election: true
I20260620 19:17:22.205768 5776 raft_consensus.cc:2454] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 320f089fa05f42f6be1232d14120fbad in term 0.
I20260620 19:17:22.205924 5843 leader_election.cc:302] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320f089fa05f42f6be1232d14120fbad, 9ce6193df1fd40e7841aa615ad204db6; no voters:
I20260620 19:17:22.206058 6104 raft_consensus.cc:2781] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:22.206108 6104 raft_consensus.cc:482] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:22.206131 6104 raft_consensus.cc:3037] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.206774 6104 raft_consensus.cc:504] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.206900 6104 leader_election.cc:288] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [CANDIDATE]: Term 1 election: Requested vote from peers 9ce6193df1fd40e7841aa615ad204db6 (127.1.1.195:36775), cbdc9babef8842bc8d6d2d81db078e1b (127.1.1.193:40909)
I20260620 19:17:22.207039 6038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e75505a23143de898fbc3870f8e168" candidate_uuid: "320f089fa05f42f6be1232d14120fbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ce6193df1fd40e7841aa615ad204db6"
I20260620 19:17:22.207127 6038 raft_consensus.cc:3037] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.207083 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e75505a23143de898fbc3870f8e168" candidate_uuid: "320f089fa05f42f6be1232d14120fbad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbdc9babef8842bc8d6d2d81db078e1b"
I20260620 19:17:22.207201 5776 raft_consensus.cc:3037] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.207772 6038 raft_consensus.cc:2454] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 320f089fa05f42f6be1232d14120fbad in term 1.
I20260620 19:17:22.207907 5843 leader_election.cc:302] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320f089fa05f42f6be1232d14120fbad, 9ce6193df1fd40e7841aa615ad204db6; no voters:
I20260620 19:17:22.207938 5776 raft_consensus.cc:2454] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 320f089fa05f42f6be1232d14120fbad in term 1.
I20260620 19:17:22.207973 6104 raft_consensus.cc:2781] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:22.208086 6104 raft_consensus.cc:686] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 1 LEADER]: Becoming Leader. State: Replica: 320f089fa05f42f6be1232d14120fbad, State: Running, Role: LEADER
I20260620 19:17:22.208180 6104 consensus_queue.cc:237] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } }
I20260620 19:17:22.208901 5634 catalog_manager.cc:5495] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad reported cstate change: term changed from 0 to 1, leader changed from <none> to 320f089fa05f42f6be1232d14120fbad (127.1.1.194). New cstate: current_term: 1 leader_uuid: "320f089fa05f42f6be1232d14120fbad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "320f089fa05f42f6be1232d14120fbad" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 41145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:22.245764 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.244864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.1.1.194:41145)
0620 19:17:22.244915 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.244917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.244925 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:22.244990 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.244991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.245002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.245044 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.245533 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.245535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.245535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.245539 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.245551 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.245555 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.245596 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.245691 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.245703 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:17:22.247277 5776 raft_consensus.cc:1261] T 37e75505a23143de898fbc3870f8e168 P cbdc9babef8842bc8d6d2d81db078e1b [term 1 FOLLOWER]: Refusing update from remote peer 320f089fa05f42f6be1232d14120fbad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:22.247277 6038 raft_consensus.cc:1261] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 1 FOLLOWER]: Refusing update from remote peer 320f089fa05f42f6be1232d14120fbad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:22.247594 6104 consensus_queue.cc:1035] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ce6193df1fd40e7841aa615ad204db6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 36775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:22.247735 6108 consensus_queue.cc:1035] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbdc9babef8842bc8d6d2d81db078e1b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:22.250603 6113 mvcc.cc:205] Tried to move back new op lower bound from 7299002541042454528 to 7299002540885266432. Current Snapshot: MvccSnapshot[applied={T|T < 7299002541042454528 or (T in {7299002541042454528})}]
I20260620 19:17:22.250763 6112 mvcc.cc:205] Tried to move back new op lower bound from 7299002541042454528 to 7299002540885266432. Current Snapshot: MvccSnapshot[applied={T|T < 7299002541042454528 or (T in {7299002541042454528})}]
I20260620 19:17:22.252061 5633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:22.252187 5633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:22.252924 5633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ba2f75b78d84d01a7092b0941613040: Sending DeleteTablet for 3 replicas of tablet 37e75505a23143de898fbc3870f8e168
I20260620 19:17:22.253242 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e75505a23143de898fbc3870f8e168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:22 UTC) from {username='slave'} at 127.0.0.1:56562
I20260620 19:17:22.253242 5887 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e75505a23143de898fbc3870f8e168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:22 UTC) from {username='slave'} at 127.0.0.1:33584
I20260620 19:17:22.253460 6122 tablet_replica.cc:330] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: stopping tablet replica
I20260620 19:17:22.253443 6018 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e75505a23143de898fbc3870f8e168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:22 UTC) from {username='slave'} at 127.0.0.1:38742
I20260620 19:17:22.253551 6122 raft_consensus.cc:2229] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:22.253626 6122 raft_consensus.cc:2258] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:22.253630 6123 tablet_replica.cc:330] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: stopping tablet replica
I20260620 19:17:22.253708 6123 raft_consensus.cc:2229] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:22.253855 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5694
I20260620 19:17:22.254002 6122 ts_tablet_manager.cc:1905] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:22.255421 6122 ts_tablet_manager.cc:1918] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:22.255489 6122 log.cc:1196] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/37e75505a23143de898fbc3870f8e168
I20260620 19:17:22.255741 6122 ts_tablet_manager.cc:1939] T 37e75505a23143de898fbc3870f8e168 P 320f089fa05f42f6be1232d14120fbad: Deleting consensus metadata
I20260620 19:17:22.256112 5621 catalog_manager.cc:4841] TS 320f089fa05f42f6be1232d14120fbad (127.1.1.194:41145): tablet 37e75505a23143de898fbc3870f8e168 (table test-table [id=2621b6aaf5374c2592b120234074bd67]) successfully deleted
I20260620 19:17:22.257176 6123 raft_consensus.cc:2258] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:22.257579 6123 ts_tablet_manager.cc:1905] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:22.258975 6123 ts_tablet_manager.cc:1918] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:22.259045 6123 log.cc:1196] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/37e75505a23143de898fbc3870f8e168
I20260620 19:17:22.259274 6123 ts_tablet_manager.cc:1939] T 37e75505a23143de898fbc3870f8e168 P 9ce6193df1fd40e7841aa615ad204db6: Deleting consensus metadata
I20260620 19:17:22.259673 5621 catalog_manager.cc:4841] TS 9ce6193df1fd40e7841aa615ad204db6 (127.1.1.195:36775): tablet 37e75505a23143de898fbc3870f8e168 (table test-table [id=2621b6aaf5374c2592b120234074bd67]) successfully deleted
W20260620 19:17:22.260125 5619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:40909 (error 108)
I20260620 19:17:22.260318 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5825
I20260620 19:17:22.260983 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.260446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40909 (local address 127.0.0.1:56578)
0620 19:17:22.260923 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.260961 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
W20260620 19:17:22.261111 5619 catalog_manager.cc:4584] TS cbdc9babef8842bc8d6d2d81db078e1b (127.1.1.193:40909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37e75505a23143de898fbc3870f8e168: Network error: Client connection negotiation failed: client connection to 127.1.1.193:40909: connect: Connection refused (error 111)
I20260620 19:17:22.265098 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5956
I20260620 19:17:22.268640 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 5603
2026-06-20T19:17:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (741 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:17:22.286861 1031 mini_oidc.cc:152] Starting JWKS server
I20260620 19:17:22.287792 1031 webserver.cc:460] Webserver started at https://127.0.0.1:46259/ using document root <none> and password file <none>
I20260620 19:17:22.287886 1031 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:17:22.288061 1031 webserver.cc:460] Webserver started at http://127.0.0.1:41085/ using document root <none> and password file <none>
2026-06-20T19:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:22Z Disabled control of system clock
I20260620 19:17:22.294651 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34977
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:34977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/testchainca.cert with env {}
W20260620 19:17:22.366393 6130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:22.366564 6130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:22.366585 6130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:22.367547 6130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:22.367575 6130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:22.367587 6130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:22.367599 6130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:22.367621 6130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:17:22.367631 6130 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:17:22.367641 6130 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46259/jwks.json
I20260620 19:17:22.368901 6130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:34977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:22.369055 6130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:22.369259 6130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:22.373401 6135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.373414 6138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.373485 6130 server_base.cc:1034] running on GCE node
W20260620 19:17:22.373414 6136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.373818 6130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:22.374107 6130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:22.375259 6130 hybrid_clock.cc:648] HybridClock initialized: now 1781983042375251 us; error 37 us; skew 500 ppm
I20260620 19:17:22.376356 6130 webserver.cc:460] Webserver started at http://127.1.1.254:37717/ using document root <none> and password file <none>
I20260620 19:17:22.376590 6130 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:22.376654 6130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:22.376782 6130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:22.377785 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "161db4d3ad694ff98c60ee12976c3a68"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "77398a22d28eea17c1eb0c782a93a6bc"
server_key_iv: "faf9097461c50fcc1004a6fe512cd929"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.378121 6130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "161db4d3ad694ff98c60ee12976c3a68"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "77398a22d28eea17c1eb0c782a93a6bc"
server_key_iv: "faf9097461c50fcc1004a6fe512cd929"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.379259 6130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:22.379949 6144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.380115 6130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:22.380210 6130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "161db4d3ad694ff98c60ee12976c3a68"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "77398a22d28eea17c1eb0c782a93a6bc"
server_key_iv: "faf9097461c50fcc1004a6fe512cd929"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.380290 6130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:22.401574 6130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:22.401837 6130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:22.401985 6130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:22.405373 6130 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:34977
I20260620 19:17:22.405417 6196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:34977 every 8 connection(s)
I20260620 19:17:22.405720 6130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:22.406571 6197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.407998 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6130
I20260620 19:17:22.408102 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:22.408291 1031 external_mini_cluster.cc:1428] Setting key 5d13a008f8a4c03debc1265200b98c96
I20260620 19:17:22.408811 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68: Bootstrap starting.
I20260620 19:17:22.409457 6197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.409759 6197 log.cc:824] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.410411 6197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68: No bootstrap required, opened a new log
I20260620 19:17:22.412401 6197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } }
I20260620 19:17:22.412567 6197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.412603 6197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161db4d3ad694ff98c60ee12976c3a68, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.412757 6197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } }
I20260620 19:17:22.412835 6197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:22.412856 6197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:22.412917 6197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.413657 6197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } }
I20260620 19:17:22.413759 6197 leader_election.cc:302] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161db4d3ad694ff98c60ee12976c3a68; no voters:
I20260620 19:17:22.413851 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.408710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.1.1.254:34977)
0620 19:17:22.408896 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.408904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.408932 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:17:22.409473 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.409481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.409553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.409694 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.409848 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.410998 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.411743 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.412410 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.412530 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.413189 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.413193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.413193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.413404 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.413444 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.413456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.413520 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.413574 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.413645 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260620 19:17:22.413894 6197 leader_election.cc:288] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:22.413972 6202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:22.414088 6202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [term 1 LEADER]: Becoming Leader. State: Replica: 161db4d3ad694ff98c60ee12976c3a68, State: Running, Role: LEADER
I20260620 19:17:22.414095 6197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:22.414191 6202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } }
I20260620 19:17:22.414505 6202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161db4d3ad694ff98c60ee12976c3a68. Latest consensus state: current_term: 1 leader_uuid: "161db4d3ad694ff98c60ee12976c3a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } } }
I20260620 19:17:22.414549 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:17:22.414573 6202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:22.414549 6203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161db4d3ad694ff98c60ee12976c3a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161db4d3ad694ff98c60ee12976c3a68" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34977 } } }
I20260620 19:17:22.414613 6203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:22.415303 6206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:22.415722 6206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:22.417032 6206 catalog_manager.cc:1347] Generated new cluster ID: 46ce3153d8f247da8335bdf46a259447
I20260620 19:17:22.417117 6206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:22.423395 6206 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:22.423816 6206 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:22.429949 6206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161db4d3ad694ff98c60ee12976c3a68: Generated new TSK 0
I20260620 19:17:22.430102 6206 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:17:22.457094 6227 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:17:22.457159 6227 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:17:22.457302 6200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.436538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.1.1.254:34977)
0620 19:17:22.436573 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.436574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.436681 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:17:22.436766 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.436766 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.436773 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.436820 (+ 47us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:17:22.436929 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.437377 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.438365 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.438525 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.438565 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.438779 (+ 214us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:17:22.457129 (+ 18350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:17:22.457254 (+ 125us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:17:22.457363 6200 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48424: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:17:22.457597 6226 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:22.436466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34977 (local address 127.0.0.1:48424)
0620 19:17:22.436630 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.436640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.436697 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.436833 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.436834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.436836 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:17:22.436877 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.436879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.437444 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.437445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.438286 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.438288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.438585 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.438591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.438672 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.438691 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:17:22.457279 (+ 18588us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:17:22.457370 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.1.254:34977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.1.254:34977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:17:22.457836 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6130
2026-06-20T19:17:22Z chronyd exiting
I20260620 19:17:22.493283 1031 test_util.cc:175] -----------------------------------------------
I20260620 19:17:22.493330 1031 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781983021660275-1031-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (207 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:17:22.493922 1031 mini_oidc.cc:152] Starting JWKS server
I20260620 19:17:22.494722 1031 webserver.cc:460] Webserver started at https://127.0.0.1:41615/ using document root <none> and password file <none>
I20260620 19:17:22.494776 1031 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:17:22.494926 1031 webserver.cc:460] Webserver started at http://127.0.0.1:42459/ using document root <none> and password file <none>
2026-06-20T19:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:22Z Disabled control of system clock
I20260620 19:17:22.501596 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44491
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:40579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44491
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41615/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/testchainca.cert with env {}
W20260620 19:17:22.573058 6233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:22.573266 6233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:22.573302 6233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:22.574339 6233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:22.574395 6233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:22.574416 6233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:22.574469 6233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:22.574499 6233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:17:22.574523 6233 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:17:22.574548 6233 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41615/jwks.json
I20260620 19:17:22.575817 6233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:40579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41615/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:22.576040 6233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:22.576227 6233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:22.580642 6239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.580684 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.580737 6233 server_base.cc:1034] running on GCE node
W20260620 19:17:22.580642 6238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.581197 6233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:22.581482 6233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:22.582628 6233 hybrid_clock.cc:648] HybridClock initialized: now 1781983042582620 us; error 26 us; skew 500 ppm
I20260620 19:17:22.583761 6233 webserver.cc:460] Webserver started at http://127.1.1.254:35083/ using document root <none> and password file <none>
I20260620 19:17:22.584018 6233 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:22.584064 6233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:22.584213 6233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:22.585182 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "3ac47e73b37c4959bb1cf37f97bb2314"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "2bedabaa17c746de87293c808985d94e"
server_key_iv: "5a29932a08f601fc8d00476de92b62cd"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.585489 6233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "3ac47e73b37c4959bb1cf37f97bb2314"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "2bedabaa17c746de87293c808985d94e"
server_key_iv: "5a29932a08f601fc8d00476de92b62cd"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.586593 6233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:22.587188 6247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.587387 6233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:22.587500 6233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "3ac47e73b37c4959bb1cf37f97bb2314"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "2bedabaa17c746de87293c808985d94e"
server_key_iv: "5a29932a08f601fc8d00476de92b62cd"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.587565 6233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:22.609944 6233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:22.610170 6233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:22.610399 6233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:22.614048 6233 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44491
I20260620 19:17:22.614076 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44491 every 8 connection(s)
I20260620 19:17:22.614384 6233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:22.615188 6300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.615466 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6233
I20260620 19:17:22.615558 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:22.615725 1031 external_mini_cluster.cc:1428] Setting key 01c781803ded6cf4ad0316aaa3aff364
I20260620 19:17:22.618047 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314: Bootstrap starting.
I20260620 19:17:22.618646 6300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.618922 6300 log.cc:824] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.619556 6300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314: No bootstrap required, opened a new log
I20260620 19:17:22.620613 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.616147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.1.1.254:44491)
0620 19:17:22.616317 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.616324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.616344 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:17:22.616705 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.616710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.616760 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.616858 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.617112 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.617885 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.618871 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.619255 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.619381 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.620063 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.620067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.620067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.620215 (+ 148us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.620243 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.620252 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.620317 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.620397 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.620470 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260620 19:17:22.621508 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:17:22.621608 6300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } }
I20260620 19:17:22.621757 6300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.621801 6300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac47e73b37c4959bb1cf37f97bb2314, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.621910 6300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } }
I20260620 19:17:22.622000 6300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:22.622042 6300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:22.622089 6300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.622586 6300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } }
I20260620 19:17:22.622713 6300 leader_election.cc:302] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ac47e73b37c4959bb1cf37f97bb2314; no voters:
I20260620 19:17:22.622861 6300 leader_election.cc:288] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:22.622884 6305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:22.622997 6305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac47e73b37c4959bb1cf37f97bb2314, State: Running, Role: LEADER
I20260620 19:17:22.623072 6305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } }
I20260620 19:17:22.623073 6300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:22.623423 6306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } } }
I20260620 19:17:22.623421 6307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ac47e73b37c4959bb1cf37f97bb2314. Latest consensus state: current_term: 1 leader_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ac47e73b37c4959bb1cf37f97bb2314" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44491 } } }
I20260620 19:17:22.623530 6306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:22.623538 6307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:22.623737 6311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:22.624210 6311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:22.625753 6311 catalog_manager.cc:1347] Generated new cluster ID: 113e2b10b3ed4afd8aab26ed88299a8f
I20260620 19:17:22.625801 6311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:22.634231 6311 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:22.634876 6311 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:22.638247 6311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ac47e73b37c4959bb1cf37f97bb2314: Generated new TSK 0
I20260620 19:17:22.638377 6311 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:17:22.661962 6330 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:17:22.662019 6330 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:17:22.662236 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.642336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48932 (local address 127.1.1.254:44491)
0620 19:17:22.642384 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.642385 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.642511 (+ 126us) server_negotiation.cc:407] Connection header received
0620 19:17:22.642589 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.642590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.642607 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.642719 (+ 112us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:17:22.642819 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.643233 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.643930 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.644136 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.644199 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.644360 (+ 161us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:17:22.662075 (+ 17715us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:17:22.662196 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 19:17:22.662297 6303 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48932: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:17:22.662357 6329 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:22.642272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44491 (local address 127.0.0.1:48932)
0620 19:17:22.642445 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.642456 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.642527 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.642720 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.642721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.642722 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:17:22.642768 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.642770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.643324 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.643325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.643848 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.643849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.644210 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.644216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.644265 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.644270 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:17:22.662173 (+ 17903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:17:22.662267 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.1.254:44491: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260620 19:17:22.662586 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6233
2026-06-20T19:17:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:17:22.699532 1031 mini_oidc.cc:152] Starting JWKS server
I20260620 19:17:22.700170 1031 webserver.cc:460] Webserver started at https://127.0.0.1:36975/ using document root <none> and password file <none>
I20260620 19:17:22.700212 1031 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:17:22.700380 1031 webserver.cc:460] Webserver started at http://127.0.0.1:37769/ using document root <none> and password file <none>
2026-06-20T19:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:22Z Disabled control of system clock
I20260620 19:17:22.707346 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35639
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:35639
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36975/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:17:22.779115 6336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:22.779275 6336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:22.779294 6336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:22.780349 6336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:22.780383 6336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:22.780395 6336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:22.780408 6336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:22.780421 6336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:17:22.780432 6336 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:17:22.780450 6336 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36975/jwks.json
I20260620 19:17:22.781716 6336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:35639
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36975/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:22.781876 6336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:22.782068 6336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:22.786473 6342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.786494 6341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.786535 6344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.786574 6336 server_base.cc:1034] running on GCE node
I20260620 19:17:22.786824 6336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:22.787101 6336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:22.788249 6336 hybrid_clock.cc:648] HybridClock initialized: now 1781983042788230 us; error 31 us; skew 500 ppm
I20260620 19:17:22.789283 6336 webserver.cc:460] Webserver started at http://127.1.1.254:39129/ using document root <none> and password file <none>
I20260620 19:17:22.789479 6336 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:22.789538 6336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:22.789667 6336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:22.790548 6336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "4a072388e5de443d852ae334907fb4a7"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "5c86d07a8a112548876370eef151d5f1"
server_key_iv: "4c34a70f0d2b8bd6c557aeee9d0d8bb2"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.790856 6336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "4a072388e5de443d852ae334907fb4a7"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "5c86d07a8a112548876370eef151d5f1"
server_key_iv: "4c34a70f0d2b8bd6c557aeee9d0d8bb2"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.792014 6336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:22.792675 6350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.792824 6336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:22.792912 6336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "4a072388e5de443d852ae334907fb4a7"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "5c86d07a8a112548876370eef151d5f1"
server_key_iv: "4c34a70f0d2b8bd6c557aeee9d0d8bb2"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.792992 6336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:22.800360 6336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:22.800586 6336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:22.800735 6336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:22.804210 6336 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:35639
I20260620 19:17:22.804236 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:35639 every 8 connection(s)
I20260620 19:17:22.804502 6336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:22.805027 6403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:22.807202 6403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: Bootstrap starting.
I20260620 19:17:22.807638 6403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:22.807845 6403 log.cc:824] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:22.808388 6403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: No bootstrap required, opened a new log
I20260620 19:17:22.810171 6403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } }
I20260620 19:17:22.810284 6403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:22.810304 6403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a072388e5de443d852ae334907fb4a7, State: Initialized, Role: FOLLOWER
I20260620 19:17:22.810412 6403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } }
I20260620 19:17:22.810479 6403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:22.810521 6403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:22.810549 6403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:22.810652 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6336
I20260620 19:17:22.810745 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:22.810894 1031 external_mini_cluster.cc:1428] Setting key 76acfa50a03b0f62ad495ac4db7bffdb
I20260620 19:17:22.810962 6403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } }
I20260620 19:17:22.811036 6403 leader_election.cc:302] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a072388e5de443d852ae334907fb4a7; no voters:
I20260620 19:17:22.811182 6403 leader_election.cc:288] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:22.811211 6406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:22.811309 6406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 4a072388e5de443d852ae334907fb4a7, State: Running, Role: LEADER
I20260620 19:17:22.811403 6403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:22.811396 6406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } }
I20260620 19:17:22.811757 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a072388e5de443d852ae334907fb4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } } }
I20260620 19:17:22.811838 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:22.811964 6406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a072388e5de443d852ae334907fb4a7. Latest consensus state: current_term: 1 leader_uuid: "4a072388e5de443d852ae334907fb4a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a072388e5de443d852ae334907fb4a7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35639 } } }
I20260620 19:17:22.812033 6406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7 [sys.catalog]: This master's current role is: LEADER
W20260620 19:17:22.813223 6424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:17:22.813289 6424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:17:22.813342 6414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:22.813520 6414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:22.814723 6414 catalog_manager.cc:1347] Generated new cluster ID: 028740fff58d4ce1bfea44a1bfccac5c
I20260620 19:17:22.814770 6414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:22.816481 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.811532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.1.1.254:35639)
0620 19:17:22.811701 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.811708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.811739 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:17:22.811934 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.811951 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.812021 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.812236 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:22.812445 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.813700 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.814678 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.815206 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.815307 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.815857 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:22.815860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:22.815861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:22.816043 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:22.816080 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:22.816092 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:22.816153 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:22.816230 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:17:22.816300 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260620 19:17:22.817324 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:17:22.822331 6414 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:22.822769 6414 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:22.825824 6414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a072388e5de443d852ae334907fb4a7: Generated new TSK 0
I20260620 19:17:22.825978 6414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:17:22.836557 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:22.833287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.1.1.254:35639)
0620 19:17:22.833332 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:22.833334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:22.833423 (+ 89us) server_negotiation.cc:407] Connection header received
0620 19:17:22.833496 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:22.833497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:22.833505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:22.833546 (+ 41us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:17:22.833645 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.834134 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.835225 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.835393 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.835452 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.835609 (+ 157us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:17:22.836335 (+ 726us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36975/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:17:22.836483 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36975/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:17:22.836606 6410 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36975/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 19:17:22.836637 6432 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:22.833192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35639 (local address 127.0.0.1:60116)
0620 19:17:22.833371 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:22.833380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:22.833436 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:22.833547 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:22.833548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:22.833549 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:17:22.833593 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.833595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.834227 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.834228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.835147 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:22.835149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:22.835471 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:22.835475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:22.835530 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:22.835535 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:17:22.836434 (+ 899us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36975/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:17:22.836519 (+ 85us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.1.254:35639: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36975/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260620 19:17:22.836910 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6336
E20260620 19:17:22.847886 6433 webserver.cc:533] Webserver: sslize failed:
E20260620 19:17:22.847934 6433 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:17:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (175 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:22Z Disabled control of system clock
I20260620 19:17:22.881024 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37525
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:33535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:37525
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/testchainca.cert with env {}
W20260620 19:17:22.954921 6437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:22.955205 6437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:22.955248 6437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:22.956609 6437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:22.956645 6437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:22.956656 6437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:22.956667 6437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:22.956678 6437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:17:22.956727 6437 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:17:22.956740 6437 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:17:22.958245 6437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:37525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:22.958436 6437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:22.958637 6437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:22.963150 6443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.963166 6445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:22.963197 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:22.963323 6437 server_base.cc:1034] running on GCE node
I20260620 19:17:22.963532 6437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:22.963790 6437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:22.964938 6437 hybrid_clock.cc:648] HybridClock initialized: now 1781983042964919 us; error 34 us; skew 500 ppm
I20260620 19:17:22.965981 6437 webserver.cc:460] Webserver started at http://127.1.1.254:43471/ using document root <none> and password file <none>
I20260620 19:17:22.966181 6437 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:22.966223 6437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:22.966349 6437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:22.967307 6437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "11e184cf50ae4ec99d2b9c17938a1638"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "17813d671bb59bf3ccc6c07e2e731071"
server_key_iv: "e1de206abed1208e65fbf4086de6157b"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.967620 6437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "11e184cf50ae4ec99d2b9c17938a1638"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "17813d671bb59bf3ccc6c07e2e731071"
server_key_iv: "e1de206abed1208e65fbf4086de6157b"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.968743 6437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:22.969419 6451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.969570 6437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:22.969660 6437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "11e184cf50ae4ec99d2b9c17938a1638"
format_stamp: "Formatted at 2026-06-20 19:17:22 on dist-test-slave-mwkb"
server_key: "17813d671bb59bf3ccc6c07e2e731071"
server_key_iv: "e1de206abed1208e65fbf4086de6157b"
server_key_version: "encryptionkey@0"
I20260620 19:17:22.969729 6437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:22.995846 6437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:22.996109 6437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:22.996251 6437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:23.000078 6437 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:37525
I20260620 19:17:23.000110 6503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:37525 every 8 connection(s)
I20260620 19:17:23.000414 6437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:23.001348 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:23.003305 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638: Bootstrap starting.
I20260620 19:17:23.003782 6504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:23.004021 6504 log.cc:824] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:23.004588 6504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638: No bootstrap required, opened a new log
I20260620 19:17:23.004789 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6437
I20260620 19:17:23.004891 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:23.005053 1031 external_mini_cluster.cc:1428] Setting key 3dab174d319fb1d9e6ecea5404593a5b
I20260620 19:17:23.006429 6504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } }
I20260620 19:17:23.006578 6504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:23.006621 6504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11e184cf50ae4ec99d2b9c17938a1638, State: Initialized, Role: FOLLOWER
I20260620 19:17:23.006742 6504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } }
I20260620 19:17:23.006831 6504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:23.006863 6504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:23.006908 6504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:23.007391 6504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } }
I20260620 19:17:23.007467 6504 leader_election.cc:302] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11e184cf50ae4ec99d2b9c17938a1638; no voters:
I20260620 19:17:23.007618 6504 leader_election.cc:288] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:23.007651 6509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:23.007751 6509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [term 1 LEADER]: Becoming Leader. State: Replica: 11e184cf50ae4ec99d2b9c17938a1638, State: Running, Role: LEADER
I20260620 19:17:23.007814 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } }
I20260620 19:17:23.007838 6504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:23.008250 6510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11e184cf50ae4ec99d2b9c17938a1638. Latest consensus state: current_term: 1 leader_uuid: "11e184cf50ae4ec99d2b9c17938a1638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } } }
I20260620 19:17:23.008320 6510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:23.008522 6514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:23.009298 6514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:23.009735 6511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11e184cf50ae4ec99d2b9c17938a1638" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11e184cf50ae4ec99d2b9c17938a1638" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37525 } } }
I20260620 19:17:23.009856 6511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:23.010939 6514 catalog_manager.cc:1347] Generated new cluster ID: 5426eb90edba4b6999dc0c215063962f
I20260620 19:17:23.010987 6514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:23.011754 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:23.005528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.1.1.254:37525)
0620 19:17:23.005703 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:23.005709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:23.005730 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:17:23.006419 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:23.006425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:23.006476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:23.006615 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:23.006811 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.008281 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.009545 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.010187 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.010617 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:23.010963 (+ 346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:23.010966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:23.010966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:17:23.011173 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:23.011211 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:17:23.011224 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:23.011282 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:23.011393 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:17:23.011561 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260620 19:17:23.012506 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:17:23.017359 6514 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:23.017827 6514 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:23.024156 6514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11e184cf50ae4ec99d2b9c17938a1638: Generated new TSK 0
I20260620 19:17:23.024308 6514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:17:23.081081 6508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:23.032294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38070 (local address 127.1.1.254:37525)
0620 19:17:23.032356 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:23.032358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:23.032478 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:17:23.032550 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:23.032551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:23.032562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:23.032642 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:17:23.032758 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.033263 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.033911 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.034116 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.034185 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:23.034352 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:17:23.080831 (+ 46479us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:17:23.080988 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38070: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 19:17:23.081146 6508 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38070: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 19:17:23.081179 6533 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:23.032225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37525 (local address 127.0.0.1:38070)
0620 19:17:23.032401 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:23.032410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:23.032493 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:23.032641 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:23.032642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:23.032643 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:17:23.032702 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:23.032703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.033339 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.033340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:23.033833 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:23.033834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:23.034183 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:23.034186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:23.034245 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:23.034249 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:17:23.080942 (+ 46693us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:17:23.081035 (+ 93us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.1.254:37525: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260620 19:17:23.081437 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6437
2026-06-20T19:17:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (223 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (78 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (79 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:23 dist-test-slave-mwkb krb5kdc[6545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:23 dist-test-slave-mwkb krb5kdc[6545](info): set up 2 sockets
Jun 20 19:17:23 dist-test-slave-mwkb krb5kdc[6545](info): commencing operation
krb5kdc: starting...
W20260620 19:17:24.196914 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:17:24.297385 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46043
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:24.370890 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:24.371076 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:24.371110 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:24.372208 6558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:24.372270 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:24.372294 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:24.372320 6558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:24.372344 6558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:24.373764 6558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:24.373997 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:24.374208 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:24.378497 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.378481 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.378496 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:24.378532 6558 server_base.cc:1034] running on GCE node
I20260620 19:17:24.378829 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:24.379061 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:24.380215 6558 hybrid_clock.cc:648] HybridClock initialized: now 1781983044380198 us; error 28 us; skew 500 ppm
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:24.382413 6558 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:17:24.382846 6558 webserver.cc:460] Webserver started at http://127.1.1.254:44863/ using document root <none> and password file <none>
I20260620 19:17:24.383069 6558 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:24.383128 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:24.383260 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:24.384182 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "05ebe7b7ef214493b2e020b43b126cd7"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5e305e2a26fa5a5f517f87ec5cbbec81"
server_key_iv: "4db0eae799ff734a8c8196f6d6c4091a"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.384491 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "05ebe7b7ef214493b2e020b43b126cd7"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5e305e2a26fa5a5f517f87ec5cbbec81"
server_key_iv: "4db0eae799ff734a8c8196f6d6c4091a"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.385686 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:24.386335 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.386478 6558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:24.386566 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "05ebe7b7ef214493b2e020b43b126cd7"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5e305e2a26fa5a5f517f87ec5cbbec81"
server_key_iv: "4db0eae799ff734a8c8196f6d6c4091a"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.386637 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:24.392987 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:24.394646 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:24.394791 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:24.398444 6558 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46043
I20260620 19:17:24.398470 6625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46043 every 8 connection(s)
I20260620 19:17:24.398747 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:24.399535 6626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:24.400692 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6558
I20260620 19:17:24.400796 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:24.400974 1031 external_mini_cluster.cc:1428] Setting key 741a74000cd070757b55adc67691c6ab
I20260620 19:17:24.401971 6626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7: Bootstrap starting.
I20260620 19:17:24.402657 6626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:24.402995 6626 log.cc:824] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:24.403620 6626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7: No bootstrap required, opened a new log
I20260620 19:17:24.405659 6626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } }
I20260620 19:17:24.405828 6626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:24.405862 6626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ebe7b7ef214493b2e020b43b126cd7, State: Initialized, Role: FOLLOWER
I20260620 19:17:24.405984 6626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } }
I20260620 19:17:24.406083 6626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:24.406116 6626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:24.406163 6626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:24.406839 6626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } }
I20260620 19:17:24.406955 6626 leader_election.cc:302] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ebe7b7ef214493b2e020b43b126cd7; no voters:
I20260620 19:17:24.407119 6626 leader_election.cc:288] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:24.407148 6631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:24.407260 6631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [term 1 LEADER]: Becoming Leader. State: Replica: 05ebe7b7ef214493b2e020b43b126cd7, State: Running, Role: LEADER
I20260620 19:17:24.407366 6626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:24.407346 6631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } }
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:24.407673 6631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ebe7b7ef214493b2e020b43b126cd7. Latest consensus state: current_term: 1 leader_uuid: "05ebe7b7ef214493b2e020b43b126cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } } }
I20260620 19:17:24.407766 6631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:24.407924 6632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ebe7b7ef214493b2e020b43b126cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ebe7b7ef214493b2e020b43b126cd7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46043 } } }
I20260620 19:17:24.407984 6632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:24.408107 6636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:24.408872 6636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:24.411763 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.401436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.1.1.254:46043)
0620 19:17:24.401651 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:24.401659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:24.401688 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:17:24.402075 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:24.402083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:24.402141 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:24.402247 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:24.403054 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.404024 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.404989 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.405387 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.405457 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.408202 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:24.408205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:24.408206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:24.408215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:24.410181 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.410541 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.410544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.410544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.410660 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.410846 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.410847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.410847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.411134 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:24.411271 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:24.411337 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:17:24.411581 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":112,"threads_started":1}
I20260620 19:17:24.413530 6636 catalog_manager.cc:1347] Generated new cluster ID: 1b5e02b9cba64e738ace38b9268e8d02
I20260620 19:17:24.413585 6636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:24.418164 6636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:24.418610 6636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:24.421298 6636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ebe7b7ef214493b2e020b43b126cd7: Generated new TSK 0
I20260620 19:17:24.421445 6636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:17:24.448689 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:24.521417 6652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:24.521595 6652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:24.521633 6652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:24.522854 6652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:24.522892 6652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:24.522917 6652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:24.524219 6652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:24.524372 6652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:24.524602 6652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:24.528977 6660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.528949 6658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.528956 6657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:24.528986 6652 server_base.cc:1034] running on GCE node
I20260620 19:17:24.529350 6652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:24.529574 6652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:24.530707 6652 hybrid_clock.cc:648] HybridClock initialized: now 1781983044530698 us; error 25 us; skew 500 ppm
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:24.532991 6652 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:17:24.533411 6652 webserver.cc:460] Webserver started at http://127.1.1.193:46339/ using document root <none> and password file <none>
I20260620 19:17:24.533617 6652 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:24.533655 6652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:24.533800 6652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:24.534667 6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "642eb0903091468c937ffb9346c95212"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5c7f023ccec3eb8039ae8f1be0b3b42a"
server_key_iv: "b85e04ab66ee8cff4dd1db64c7f72db9"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.534984 6652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "642eb0903091468c937ffb9346c95212"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5c7f023ccec3eb8039ae8f1be0b3b42a"
server_key_iv: "b85e04ab66ee8cff4dd1db64c7f72db9"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.536243 6652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:24.536954 6667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.537147 6652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:24.537250 6652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "642eb0903091468c937ffb9346c95212"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "5c7f023ccec3eb8039ae8f1be0b3b42a"
server_key_iv: "b85e04ab66ee8cff4dd1db64c7f72db9"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.537336 6652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:24.560142 6652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:24.562193 6652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:24.562362 6652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:24.562598 6652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:24.562891 6652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:24.562938 6652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.562968 6652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:24.562994 6652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.568143 6652 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:37685
I20260620 19:17:24.568214 6780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:37685 every 8 connection(s)
I20260620 19:17:24.568513 6652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:24.572592 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6652
I20260620 19:17:24.572702 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:24.572901 1031 external_mini_cluster.cc:1428] Setting key 76552816e4e9c1aa1384a531ca999e00
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:24.577481 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.569260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:47461 (local address 127.1.1.254:46043)
0620 19:17:24.569335 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:24.569337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:24.569495 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:17:24.570447 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:24.570449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:24.570461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:24.570506 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:24.571055 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.571517 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.572492 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.572744 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.572828 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.574651 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:24.574653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:24.574653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:24.574661 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:24.576004 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.576423 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.576424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.576425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.576509 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.576832 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.576833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.576833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.576931 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:24.576991 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:24.577349 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 19:17:24.577418 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:17:24.577735 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.569183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46043 (local address 127.1.1.193:47461)
0620 19:17:24.569383 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:24.569416 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:24.570326 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:24.570582 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:24.570585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:24.570801 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:24.570964 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.570968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.571603 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.571605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.572386 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.572388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.572884 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.572885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.572994 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.573395 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:24.573403 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:24.574554 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:24.576115 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.576122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.576124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.576342 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.576601 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.576602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.576602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.576755 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.577009 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:24.577016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:24.577194 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 19:17:24.577318 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 19:17:24.577492 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260620 19:17:24.578305 6781 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46043
I20260620 19:17:24.578394 6781 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:24.578574 6781 heartbeater.cc:511] Master 127.1.1.254:46043 requested a full tablet report, sending...
I20260620 19:17:24.579051 6590 ts_manager.cc:194] Registered new tserver with Master: 642eb0903091468c937ffb9346c95212 (127.1.1.193:37685)
I20260620 19:17:24.579658 6590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:47461
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:17:24.594939 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:24.669255 6786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:24.669410 6786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:24.669430 6786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:24.670367 6786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:24.670399 6786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:24.670423 6786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:24.671636 6786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:24.671799 6786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:24.671964 6786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:24.676429 6791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.676478 6792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:24.676491 6786 server_base.cc:1034] running on GCE node
W20260620 19:17:24.676597 6794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:24.676836 6786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:24.677016 6786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:24.678143 6786 hybrid_clock.cc:648] HybridClock initialized: now 1781983044678129 us; error 27 us; skew 500 ppm
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:24.680351 6786 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:17:24.680745 6786 webserver.cc:460] Webserver started at http://127.1.1.194:43387/ using document root <none> and password file <none>
I20260620 19:17:24.680959 6786 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:24.681003 6786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:24.681167 6786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:24.682051 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "4f25b6e8a4df4858bbf3404909a2c8da"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "847be0c42b12c4e994841b0eee78fe9b"
server_key_iv: "4127f694016f997efb963d8f686d2e45"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.682341 6786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "4f25b6e8a4df4858bbf3404909a2c8da"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "847be0c42b12c4e994841b0eee78fe9b"
server_key_iv: "4127f694016f997efb963d8f686d2e45"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.683509 6786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:17:24.684098 6801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.684242 6786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.684319 6786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "4f25b6e8a4df4858bbf3404909a2c8da"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "847be0c42b12c4e994841b0eee78fe9b"
server_key_iv: "4127f694016f997efb963d8f686d2e45"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.684368 6786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:24.706110 6786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:24.708171 6786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:24.708362 6786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:24.708564 6786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:24.708830 6786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:24.708880 6786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.708917 6786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:24.708937 6786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.714187 6786 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36029
I20260620 19:17:24.714246 6914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36029 every 8 connection(s)
I20260620 19:17:24.714522 6786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:17:24.718686 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6786
I20260620 19:17:24.718763 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:24.718966 1031 external_mini_cluster.cc:1428] Setting key ae51caee0138eec3beae3124c452d4b1
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:24.723311 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.715118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:44233 (local address 127.1.1.254:46043)
0620 19:17:24.715156 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:24.715157 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:24.715345 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:17:24.716151 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:24.716152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:24.716162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:24.716209 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:24.716715 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.717262 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.718218 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.718413 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.718493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.720585 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:24.720587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:24.720587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:24.720594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:24.722006 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.722419 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.722420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.722421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.722485 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.722735 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.722736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.722736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.722856 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:24.722926 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:24.723166 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:17:24.723232 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:17:24.723466 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.715088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46043 (local address 127.1.1.194:44233)
0620 19:17:24.715247 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:24.715275 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:24.716053 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:24.716265 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:24.716268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:24.716476 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:24.716625 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.716628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.717359 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.717360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.718123 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.718124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.718506 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.718506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.718618 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.719098 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:24.719107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:24.720494 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:24.722107 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.722114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.722116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.722328 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.722556 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.722557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.722557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.722665 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.722918 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:24.722923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:24.723056 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 19:17:24.723147 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:17:24.723275 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260620 19:17:24.723857 6915 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46043
I20260620 19:17:24.723922 6915 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:24.724072 6915 heartbeater.cc:511] Master 127.1.1.254:46043 requested a full tablet report, sending...
I20260620 19:17:24.724412 6590 ts_manager.cc:194] Registered new tserver with Master: 4f25b6e8a4df4858bbf3404909a2c8da (127.1.1.194:36029)
I20260620 19:17:24.724740 6590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:44233
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:17:24.741072 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:24.823272 6920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:24.823449 6920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:24.823472 6920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:24.824621 6920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:24.824663 6920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:24.824690 6920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:24.826160 6920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:24.826397 6920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:24.826608 6920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:24.831296 6925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.831269 6926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:24.831281 6928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:24.831383 6920 server_base.cc:1034] running on GCE node
I20260620 19:17:24.831740 6920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:24.831995 6920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:24.833125 6920 hybrid_clock.cc:648] HybridClock initialized: now 1781983044833104 us; error 35 us; skew 500 ppm
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:24.838233 6920 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:17:24.838619 6920 webserver.cc:460] Webserver started at http://127.1.1.195:41241/ using document root <none> and password file <none>
I20260620 19:17:24.838835 6920 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:24.838877 6920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:24.839015 6920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:24.840011 6920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "87baff11fe8e4e5aa055daea6a919e2a"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "45f89e67ff0a4baab2f53d8ea8033cd0"
server_key_iv: "2bf2061f5c79e52fbd0f5d080b6c06fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.840354 6920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "87baff11fe8e4e5aa055daea6a919e2a"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "45f89e67ff0a4baab2f53d8ea8033cd0"
server_key_iv: "2bf2061f5c79e52fbd0f5d080b6c06fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.841694 6920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:24.842334 6935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.842522 6920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:24.842590 6920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "87baff11fe8e4e5aa055daea6a919e2a"
format_stamp: "Formatted at 2026-06-20 19:17:24 on dist-test-slave-mwkb"
server_key: "45f89e67ff0a4baab2f53d8ea8033cd0"
server_key_iv: "2bf2061f5c79e52fbd0f5d080b6c06fe"
server_key_version: "encryptionkey@0"
I20260620 19:17:24.842633 6920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:24.855471 6920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:24.857020 6920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:24.857183 6920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:24.857344 6920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:24.857632 6920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:24.857664 6920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.857715 6920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:24.857744 6920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:24.863503 6920 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34911
I20260620 19:17:24.863595 7048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34911 every 8 connection(s)
I20260620 19:17:24.863915 6920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:24.864969 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 6920
I20260620 19:17:24.865114 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:24.865289 1031 external_mini_cluster.cc:1428] Setting key 6fd2b44dd520618098df17a4822916fa
Jun 20 19:17:24 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983044, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:17:24.873037 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.864593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:34169 (local address 127.1.1.254:46043)
0620 19:17:24.864635 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:24.864637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:24.864822 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:17:24.866028 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:24.866029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:24.866044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:24.866092 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:24.866728 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.867210 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.868190 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.868369 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.868461 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.870498 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:24.870499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:24.870499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:24.870508 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:24.871714 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.872109 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.872110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.872111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.872168 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:24.872412 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:24.872413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:24.872413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:24.872524 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:24.872596 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:24.872857 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 19:17:24.872941 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:24.873226 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:24.864564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46043 (local address 127.1.1.195:34169)
0620 19:17:24.864699 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:24.864740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:24.865922 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:24.866171 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:24.866175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:24.866464 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:24.866637 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.866641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.867314 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.867315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.868094 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:24.868095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:24.868478 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:24.868479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:24.868618 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:24.869021 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:24.869029 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:24.870405 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:24.871801 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.871807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.871809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.872021 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.872235 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:24.872236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:24.872236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:24.872342 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:24.872598 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:24.872605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:24.872732 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:17:24.872836 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:17:24.872989 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260620 19:17:24.873679 7049 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46043
I20260620 19:17:24.873755 7049 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:24.873932 7049 heartbeater.cc:511] Master 127.1.1.254:46043 requested a full tablet report, sending...
I20260620 19:17:24.874279 6590 ts_manager.cc:194] Registered new tserver with Master: 87baff11fe8e4e5aa055daea6a919e2a (127.1.1.195:34911)
I20260620 19:17:24.874588 6590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:34169
I20260620 19:17:24.877152 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:17:25.580466 6781 heartbeater.cc:503] Master 127.1.1.254:46043 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/test-admin.keytab.
W20260620 19:17:25.711371 1031 mini_kdc.cc:302] Time spent creating keytab for test-admin: real 0.834s user 0.000s sys 0.002s
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.713065 1031 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:17:25.714922 7056 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:25.714120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.1.1.193:37685)
0620 19:17:25.714310 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.714314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.714324 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:17:25.714347 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.714349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.714362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.714459 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.714837 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39934: BlockingRecv error: recv got EOF from 127.0.0.1:39934 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":58,"threads_started":1}
W20260620 19:17:25.715042 1031 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.716065 1031 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.716881 1031 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:25.724011 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:25.717441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.1.1.193:37685)
0620 19:17:25.717490 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.717492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.717515 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:17:25.717570 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.717571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.717580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.717620 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.717951 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.718614 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.719234 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.719550 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.719604 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:25.721507 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:25.721514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:25.721514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:25.721520 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:25.723070 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.723448 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.723450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.723451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.723499 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.723657 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.723658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.723658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.723763 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:25.723846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:25.723908 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:17:25.723961 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:17:25.725473 6915 heartbeater.cc:503] Master 127.1.1.254:46043 was elected leader, sending a full tablet report...
W20260620 19:17:25.725740 7056 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:25.725214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.1.1.193:37685)
0620 19:17:25.725244 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.725246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.725319 (+ 73us) server_negotiation.cc:407] Connection header received
0620 19:17:25.725358 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.725358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.725366 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.725428 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.725707 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39938: BlockingRecv error: recv got EOF from 127.0.0.1:39938 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:17:25.725832 1031 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.726661 1031 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.727424 1031 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:25.733335 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:25.727814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.1.1.193:37685)
0620 19:17:25.727861 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.727862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.727883 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:17:25.727935 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.727935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.727941 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.727975 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.728250 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.728638 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.729271 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.729478 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.729528 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:25.731262 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:25.731263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:25.731263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:25.731268 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:25.732403 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.732688 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.732689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.732689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.732733 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.732884 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.732885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.732885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.732956 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:25.733015 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:25.733162 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:17:25.733215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 19:17:25.734668 7056 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:17:25.734177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.1.1.193:37685)
0620 19:17:25.734231 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.734233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.734309 (+ 76us) server_negotiation.cc:407] Connection header received
0620 19:17:25.734343 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.734344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.734350 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.734387 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.734609 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39958: BlockingRecv error: recv got EOF from 127.0.0.1:39958 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 19:17:25.734771 1031 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781983021660275-1031-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.735666 1031 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:25.736452 1031 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:17:25 dist-test-slave-mwkb krb5kdc[6545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.193@KRBTEST.COM
I20260620 19:17:25.742611 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:25.736873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.1.1.193:37685)
0620 19:17:25.736932 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:25.736933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:25.736941 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:25.736996 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:25.736997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:25.737005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:25.737100 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:25.737434 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.737980 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.738529 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:25.738678 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:25.738722 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:25.740498 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:25.740499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:25.740499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:25.740505 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:25.741752 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.742065 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.742065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.742066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.742112 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:25.742253 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:25.742253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:25.742253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:25.742319 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:25.742409 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:25.742439 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 19:17:25.742491 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:17:25.743166 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6652
I20260620 19:17:25.748188 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6786
I20260620 19:17:25.752241 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6920
I20260620 19:17:25.756300 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 6558
2026-06-20T19:17:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2520 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:17:27 dist-test-slave-mwkb krb5kdc[7069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:17:27 dist-test-slave-mwkb krb5kdc[7069](info): set up 2 sockets
Jun 20 19:17:27 dist-test-slave-mwkb krb5kdc[7069](info): commencing operation
krb5kdc: starting...
W20260620 19:17:28.845011 1031 mini_kdc.cc:121] Time spent starting KDC: real 3.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 19:17:29.090267 1031 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.244s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 19:17:29.196463 1031 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.105s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:17:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab.
Entry for principal oryx/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab.
I20260620 19:17:29.267976 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46261
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab
--principal=oryx/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:29.345795 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:29.345995 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:29.346027 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:29.347067 7082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:29.347126 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:29.347148 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:29.347173 7082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:29.347201 7082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:29.348472 7082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab
--principal=oryx/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:29.348719 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:29.348915 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:29.353292 7090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.353358 7082 server_base.cc:1034] running on GCE node
W20260620 19:17:29.353298 7087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.353313 7088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.353713 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:29.353991 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:29.355156 7082 hybrid_clock.cc:648] HybridClock initialized: now 1781983049355137 us; error 42 us; skew 500 ppm
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:29.357558 7082 init.cc:377] Logged in from keytab as oryx/127.1.1.254@KRBTEST.COM (short username oryx)
I20260620 19:17:29.358014 7082 webserver.cc:460] Webserver started at http://127.1.1.254:35603/ using document root <none> and password file <none>
I20260620 19:17:29.358218 7082 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:29.358251 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:29.358340 7082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:29.359364 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "df95fcc3a9f6467e8cf4570c0d4d91af"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "023fec241fc2c3e98cb8135486fbe16d"
server_key_iv: "16c753f63923c0c838195d25c4ab5a04"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.359714 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "df95fcc3a9f6467e8cf4570c0d4d91af"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "023fec241fc2c3e98cb8135486fbe16d"
server_key_iv: "16c753f63923c0c838195d25c4ab5a04"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.361429 7082 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 19:17:29.362273 7097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.362483 7082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:29.362602 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "df95fcc3a9f6467e8cf4570c0d4d91af"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "023fec241fc2c3e98cb8135486fbe16d"
server_key_iv: "16c753f63923c0c838195d25c4ab5a04"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.362682 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:29.376216 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:29.377833 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:29.377974 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:29.381549 7082 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46261
I20260620 19:17:29.381573 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46261 every 8 connection(s)
I20260620 19:17:29.381870 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:17:29.382295 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7082
I20260620 19:17:29.382407 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:17:29.382591 1031 external_mini_cluster.cc:1428] Setting key 2815c60e35e8e9c3a692397eacd1cb47
I20260620 19:17:29.382992 7150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:29.386132 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Bootstrap starting.
I20260620 19:17:29.386811 7150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:29.387115 7150 log.cc:824] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:29.387907 7150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: No bootstrap required, opened a new log
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:29.393643 7150 raft_consensus.cc:348] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } }
I20260620 19:17:29.393787 7150 raft_consensus.cc:374] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:29.393811 7150 raft_consensus.cc:729] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df95fcc3a9f6467e8cf4570c0d4d91af, State: Initialized, Role: FOLLOWER
I20260620 19:17:29.393993 7150 consensus_queue.cc:260] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } }
I20260620 19:17:29.394079 7150 raft_consensus.cc:388] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:17:29.394116 7150 raft_consensus.cc:482] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:17:29.394170 7150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:29.394894 7150 raft_consensus.cc:504] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } }
I20260620 19:17:29.395027 7150 leader_election.cc:302] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df95fcc3a9f6467e8cf4570c0d4d91af; no voters:
I20260620 19:17:29.395215 7150 leader_election.cc:288] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:17:29.395259 7155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:29.395435 7155 raft_consensus.cc:686] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 LEADER]: Becoming Leader. State: Replica: df95fcc3a9f6467e8cf4570c0d4d91af, State: Running, Role: LEADER
I20260620 19:17:29.395467 7150 sys_catalog.cc:564] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:29.395629 7155 consensus_queue.cc:237] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } }
I20260620 19:17:29.396090 7155 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: New leader df95fcc3a9f6467e8cf4570c0d4d91af. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } }
I20260620 19:17:29.396236 7155 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:29.396694 7164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:17:29.397234 7164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:17:29.397704 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.383094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.1.1.254:46261)
0620 19:17:29.383276 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:29.383286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:29.383317 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:17:29.383805 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:29.383813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:29.383888 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:29.384042 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:29.385762 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.387129 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.388077 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.388484 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.388614 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.393100 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:29.393108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:29.393109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:29.393128 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:29.395752 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.396301 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.396305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.396305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.396420 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.396682 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.396683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.396684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.397010 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:29.397198 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:29.397276 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:17:29.397413 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 19:17:29.398284 7156 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } }
I20260620 19:17:29.398569 7156 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:29.398694 7164 catalog_manager.cc:1347] Generated new cluster ID: 945f86494ce8469e993f911b62ca54a7
I20260620 19:17:29.398734 7164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:17:29.416518 7164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:17:29.417260 7164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:17:29.428007 7164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Generated new TSK 0
I20260620 19:17:29.428151 7164 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
I20260620 19:17:29.451939 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:29.532801 7176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:29.532974 7176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:29.532997 7176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:29.534099 7176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:29.534140 7176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:29.534165 7176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:29.535488 7176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:29.535645 7176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:29.535791 7176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:29.540256 7181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.540391 7176 server_base.cc:1034] running on GCE node
W20260620 19:17:29.540246 7182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.540246 7184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.540683 7176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:29.540923 7176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:29.542101 7176 hybrid_clock.cc:648] HybridClock initialized: now 1781983049542078 us; error 40 us; skew 500 ppm
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:29.544790 7176 init.cc:377] Logged in from keytab as oryx/127.1.1.193@KRBTEST.COM (short username oryx)
I20260620 19:17:29.545284 7176 webserver.cc:460] Webserver started at http://127.1.1.193:37563/ using document root <none> and password file <none>
I20260620 19:17:29.545537 7176 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:29.545605 7176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:29.545735 7176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:29.546669 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "583e924f8d654681f50597ed1b9106ad"
server_key_iv: "e0a204235aeebda3f291e86d9c7b7b3a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.547020 7176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "583e924f8d654681f50597ed1b9106ad"
server_key_iv: "e0a204235aeebda3f291e86d9c7b7b3a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.548202 7176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:17:29.548810 7191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.548957 7176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:29.549058 7176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "583e924f8d654681f50597ed1b9106ad"
server_key_iv: "e0a204235aeebda3f291e86d9c7b7b3a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.549144 7176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:29.572085 7176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:29.573808 7176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:29.573966 7176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:29.574188 7176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:29.574492 7176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:29.574558 7176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.574592 7176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:29.574611 7176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.580212 7176 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:40291
I20260620 19:17:29.580257 7304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:40291 every 8 connection(s)
I20260620 19:17:29.580533 7176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:29.585937 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7176
I20260620 19:17:29.586042 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:17:29.586236 1031 external_mini_cluster.cc:1428] Setting key 7214b865a74f6cabdf2fbdc731bb2c87
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:29.590117 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.581140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:35515)
0620 19:17:29.581336 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:29.581368 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:29.581821 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:29.582064 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:29.582068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:29.582590 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:29.582754 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.582758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.583436 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.583437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.584204 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.584206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.584604 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.584606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.584722 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.585175 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:29.585183 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:29.586766 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:29.588661 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.588668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.588670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.589007 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.589253 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.589254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.589255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.589389 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.589637 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:29.589642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:29.589779 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 19:17:29.589855 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:17:29.589971 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260620 19:17:29.590129 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.581149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:35515 (local address 127.1.1.254:46261)
0620 19:17:29.581230 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:29.581233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:29.581452 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:17:29.581917 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:29.581918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:29.581929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:29.581990 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:29.582845 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.583336 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.584290 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.584487 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.584599 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.586875 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:29.586876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:29.586877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:29.586883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:29.588465 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.589123 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.589124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.589124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.589185 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.589459 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.589460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.589460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.589566 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:29.589638 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:29.589916 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:17:29.589997 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:17:29.590785 7305 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:29.590885 7305 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:29.591079 7305 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:29.591517 7114 ts_manager.cc:194] Registered new tserver with Master: 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:29.592129 7114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:35515
WARNING: no policy specified for oryx/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
I20260620 19:17:29.610008 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:29.693853 7310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:29.694088 7310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:29.694129 7310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:29.695374 7310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:29.695432 7310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:29.695477 7310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:29.697110 7310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:29.697362 7310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:29.697603 7310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:29.702409 7310 server_base.cc:1034] running on GCE node
W20260620 19:17:29.702363 7318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.702351 7315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.702344 7316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.702769 7310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:29.703002 7310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:29.704146 7310 hybrid_clock.cc:648] HybridClock initialized: now 1781983049704132 us; error 28 us; skew 500 ppm
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:29.706588 7310 init.cc:377] Logged in from keytab as oryx/127.1.1.194@KRBTEST.COM (short username oryx)
I20260620 19:17:29.706984 7310 webserver.cc:460] Webserver started at http://127.1.1.194:41905/ using document root <none> and password file <none>
I20260620 19:17:29.707202 7310 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:29.707247 7310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:29.707412 7310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:29.708410 7310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "21d9e09fb42c407d91228904eeef7dff"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "b46280f3a73a9667e5f82d61a52b2721"
server_key_iv: "76d49a0349cb727843f1d151a9b52ae4"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.708755 7310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "21d9e09fb42c407d91228904eeef7dff"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "b46280f3a73a9667e5f82d61a52b2721"
server_key_iv: "76d49a0349cb727843f1d151a9b52ae4"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.710043 7310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:17:29.710737 7325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.710911 7310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:29.710997 7310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "21d9e09fb42c407d91228904eeef7dff"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "b46280f3a73a9667e5f82d61a52b2721"
server_key_iv: "76d49a0349cb727843f1d151a9b52ae4"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.711067 7310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:29.726184 7310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:29.728291 7310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:29.728410 7310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:29.728608 7310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:29.728868 7310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:29.728916 7310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.728947 7310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:29.728991 7310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.734069 7310 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:42555
I20260620 19:17:29.734125 7438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:42555 every 8 connection(s)
I20260620 19:17:29.734391 7310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:29.741784 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.735003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:41325 (local address 127.1.1.254:46261)
0620 19:17:29.735063 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:29.735065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:29.735229 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:17:29.735613 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:29.735614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:29.735621 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:29.735665 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:29.736405 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.736837 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.737783 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.737956 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.738003 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.739702 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:29.739703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:29.739703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:29.739709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:29.740797 (+ 1088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.741146 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.741147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.741147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.741187 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.741385 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.741385 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.741386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.741462 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:29.741503 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:29.741682 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 19:17:29.741731 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:17:29.741921 7441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.734974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:41325)
0620 19:17:29.735111 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:29.735155 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:29.735540 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:29.735714 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:29.735717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:29.736189 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:29.736343 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.736346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.736941 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.736942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.737720 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.737722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.738020 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.738020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.738103 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.738457 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:29.738464 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:29.739624 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:29.740861 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.740865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.740867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.741039 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.741236 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.741237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.741237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.741325 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.741507 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:29.741511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:29.741596 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 19:17:29.741660 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:17:29.741749 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260620 19:17:29.742466 7439 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:29.742556 7439 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:29.742671 7439 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:29.742986 7114 ts_manager.cc:194] Registered new tserver with Master: 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555)
I20260620 19:17:29.743317 7114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:41325
I20260620 19:17:29.743889 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7310
I20260620 19:17:29.743978 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:17:29.744102 1031 external_mini_cluster.cc:1428] Setting key 9e48aad98d10bc4dcfd2074b8f010d0b
WARNING: no policy specified for oryx/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
I20260620 19:17:29.767534 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:29.843859 7444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:29.844013 7444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:29.844049 7444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:29.845156 7444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:29.845192 7444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:29.845252 7444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:29.846561 7444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:29.846715 7444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:29.846927 7444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:29.851163 7450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.851223 7452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:29.851164 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:29.851266 7444 server_base.cc:1034] running on GCE node
I20260620 19:17:29.851570 7444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:29.851819 7444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:29.852974 7444 hybrid_clock.cc:648] HybridClock initialized: now 1781983049852967 us; error 32 us; skew 500 ppm
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:29.861860 7444 init.cc:377] Logged in from keytab as oryx/127.1.1.195@KRBTEST.COM (short username oryx)
I20260620 19:17:29.862425 7444 webserver.cc:460] Webserver started at http://127.1.1.195:41805/ using document root <none> and password file <none>
I20260620 19:17:29.862679 7444 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:29.862761 7444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:29.862890 7444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:29.864151 7444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "e6342558c60a40249311417c9ee394bb"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "6414a78b4915c6fef1f133fcebaebe47"
server_key_iv: "3f54e40f13d360bfe117a56208bbb59a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.864454 7444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "e6342558c60a40249311417c9ee394bb"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "6414a78b4915c6fef1f133fcebaebe47"
server_key_iv: "3f54e40f13d360bfe117a56208bbb59a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.865797 7444 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:17:29.866575 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.866755 7444 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:29.866847 7444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "e6342558c60a40249311417c9ee394bb"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "6414a78b4915c6fef1f133fcebaebe47"
server_key_iv: "3f54e40f13d360bfe117a56208bbb59a"
server_key_version: "encryptionkey@0"
I20260620 19:17:29.866914 7444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:29.886646 7444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:29.888844 7444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:29.888996 7444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:29.889241 7444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:29.889539 7444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:29.889572 7444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.889593 7444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:29.889605 7444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:29.895516 7444 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:41655
I20260620 19:17:29.895596 7572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:41655 every 8 connection(s)
I20260620 19:17:29.895910 7444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:29.901446 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7444
I20260620 19:17:29.901551 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:17:29.901734 1031 external_mini_cluster.cc:1428] Setting key 4e3e8da1633fecd4dbdb19d6c184946d
Jun 20 19:17:29 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:29.905061 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.896613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:52289 (local address 127.1.1.254:46261)
0620 19:17:29.896692 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:29.896694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:29.896887 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:17:29.897435 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:29.897436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:29.897477 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:29.897555 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:29.898412 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.898996 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.899944 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.900121 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.900174 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.902312 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:29.902314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:29.902315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:29.902322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:29.903717 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.904130 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.904132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.904132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.904189 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:29.904452 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:29.904452 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:29.904453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:29.904559 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:29.904628 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:29.904895 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:17:29.904971 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:17:29.905259 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:29.896573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:52289)
0620 19:17:29.896766 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:29.896808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:29.897342 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:29.897602 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:29.897605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:29.898143 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:29.898324 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.898327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.899059 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.899060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.899870 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:29.899872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:29.900191 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:29.900191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:29.900281 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:29.900663 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:29.900671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:29.902240 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:29.903819 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.903825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.903827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.904040 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.904264 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:29.904265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:29.904265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:29.904378 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:29.904632 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:29.904638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:29.904766 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:17:29.904876 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:17:29.905023 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":110,"threads_started":1}
I20260620 19:17:29.905691 7573 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:29.905794 7573 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:29.905982 7573 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:29.906338 7114 ts_manager.cc:194] Registered new tserver with Master: e6342558c60a40249311417c9ee394bb (127.1.1.195:41655)
I20260620 19:17:29.906653 7114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:52289
I20260620 19:17:29.913434 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.1.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.253.keytab.
Entry for principal oryx/127.1.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.253.keytab.
I20260620 19:17:29.935613 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:37587
--webserver_interface=127.1.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.253.keytab
--principal=oryx/127.1.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46261,127.1.1.253:37587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:30.012094 7578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:30.012266 7578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:30.012287 7578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:30.013430 7578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:30.013468 7578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:30.013480 7578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:30.013522 7578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:30.013541 7578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:30.014868 7578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46261,127.1.1.253:37587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.253.keytab
--principal=oryx/127.1.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:37587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
W20260620 19:17:30.014968 7578 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.1.254:46261,127.1.1.253:37587'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:17:30.015132 7578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:30.015326 7578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:17:30.019831 7578 server_base.cc:1034] running on GCE node
W20260620 19:17:30.019836 7586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.019841 7583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.019987 7584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.020284 7578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.020576 7578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.021741 7578 hybrid_clock.cc:648] HybridClock initialized: now 1781983050021734 us; error 34 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.024137 7578 init.cc:377] Logged in from keytab as oryx/127.1.1.253@KRBTEST.COM (short username oryx)
I20260620 19:17:30.024530 7578 webserver.cc:460] Webserver started at http://127.1.1.253:37879/ using document root <none> and password file <none>
I20260620 19:17:30.024751 7578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.024814 7578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.024956 7578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:30.025949 7578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/instance:
uuid: "8fbca32dc1904175b6babc7a337fca1d"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "8c5bf52bdfbb4f28bf4bbc7ddfb97fbb"
server_key_iv: "0bd47f5e2fbd7985cc12f65707e3a56d"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.026286 7578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal/instance:
uuid: "8fbca32dc1904175b6babc7a337fca1d"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "8c5bf52bdfbb4f28bf4bbc7ddfb97fbb"
server_key_iv: "0bd47f5e2fbd7985cc12f65707e3a56d"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.027465 7578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:30.028080 7593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.028239 7578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:30.028322 7578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "8fbca32dc1904175b6babc7a337fca1d"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "8c5bf52bdfbb4f28bf4bbc7ddfb97fbb"
server_key_iv: "0bd47f5e2fbd7985cc12f65707e3a56d"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.028404 7578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.044539 7578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.046196 7578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:30.046285 7578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.049616 7578 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.253:37587
I20260620 19:17:30.049657 7645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.253:37587 every 8 connection(s)
I20260620 19:17:30.049942 7578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
I20260620 19:17:30.050772 7646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:30.052124 7646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.253@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:30.059904 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7578
I20260620 19:17:30.061064 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.052780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42202)
0620 19:17:30.052938 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.052974 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.053410 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.053598 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.053601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.053789 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.053934 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.053937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.054463 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.054464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.055212 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.055214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.055490 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.055490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.055592 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.056052 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.056059 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.057360 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.059034 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.059039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.059040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.059233 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.059532 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.059534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.059534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.059835 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.060339 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.060358 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.060540 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.060677 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.060849 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 19:17:30.061110 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.052801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42202 (local address 127.1.1.254:46261)
0620 19:17:30.052870 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.052871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.053050 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:17:30.053498 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.053499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.053506 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.053550 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.053988 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.054399 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.055272 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.055429 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.055475 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.057440 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.057441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.057442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.057448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.058928 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.059344 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.059345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.059345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.059413 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.059959 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.059961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.059961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.060243 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.060313 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.060871 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.060992 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:17:30.061952 7646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.253@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
I20260620 19:17:30.070534 7648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.062345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32866)
0620 19:17:30.062400 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.062409 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.062442 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.062728 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.062729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.062894 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.062937 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.062938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.063698 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.063699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.064433 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.064435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.064924 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.064925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.064985 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.065656 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.065660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.067056 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.069498 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.069499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.069499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.069689 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.069885 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.069885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.069885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.069924 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.070218 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.070218 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.070258 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.070334 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.070425 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:17:30.070683 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.062422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.1.1.253:37587)
0620 19:17:30.062580 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.062582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.062590 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:30.062607 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.062608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.062618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.062694 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.063016 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.063622 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.064513 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.064847 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.064919 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.067140 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.067146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.067146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.067153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.069411 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.069760 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.069763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.069763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.069820 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.069978 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.069979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.069980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.070143 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.070218 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.070370 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.070514 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260620 19:17:30.071439 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.060694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.1.1.253:37587)
0620 19:17:30.060855 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.060860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.061005 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:17:30.061090 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.061093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.061108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.061206 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.062361 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.063345 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.064402 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.064847 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.064919 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.067858 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.067859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.067860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.067866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.070070 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.070460 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.070461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.070462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.070515 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.070725 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.070726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.070727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.070850 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.070962 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.071306 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.071356 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 19:17:30.072098 7646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Bootstrap starting.
I20260620 19:17:30.072631 7646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:30.072911 7646 log.cc:824] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:30.073598 7646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: No bootstrap required, opened a new log
I20260620 19:17:30.074782 7646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } }
I20260620 19:17:30.074885 7646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:30.074927 7646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fbca32dc1904175b6babc7a337fca1d, State: Initialized, Role: FOLLOWER
I20260620 19:17:30.075062 7646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } }
I20260620 19:17:30.075292 7653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } } }
I20260620 19:17:30.075332 7646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:30.075408 7653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:17:30.077167 7664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:17:30.077211 7664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:17:30.078299 7578 master_runner.cc:428] Detected that this master 8fbca32dc1904175b6babc7a337fca1d is joining an existing cluster
I20260620 19:17:30.078358 7578 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.1.253:37587
I20260620 19:17:30.085749 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.079231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.1.1.254:46261)
0620 19:17:30.079270 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.079272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.079457 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:17:30.079559 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.079560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.079570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.079600 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.080108 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.080595 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081514 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.081748 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081776 (+ 28us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.082855 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.082856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.082857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.082863 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.084442 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.084875 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.084876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.084876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.084928 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.085207 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.085208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.085209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.085331 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.085377 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.085599 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.085666 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:17:30.085760 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.079204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42206)
0620 19:17:30.079352 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.079360 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.079485 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.079656 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.079657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.079916 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.080009 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.080010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.080666 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.080667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.081430 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.081432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.081834 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.081882 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.082277 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.082280 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.082755 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.084548 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.084549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.084550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.084804 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.085002 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.085002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.085002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.085124 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.085441 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.085442 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.085478 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.085570 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.085660 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260620 19:17:30.086071 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.079228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.1.1.253:37587)
0620 19:17:30.079255 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.079257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.079528 (+ 271us) server_negotiation.cc:407] Connection header received
0620 19:17:30.079572 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.079573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.079582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.079642 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.080085 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.080763 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081597 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.081876 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081941 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.082923 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.082924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.082924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.082931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.084908 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.085293 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.085294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.085294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.085351 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.085549 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.085550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.085550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.085670 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.085733 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.085882 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.085950 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:17:30.086438 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.079290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32872)
0620 19:17:30.079456 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.079465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.079517 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.079656 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.079657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.079911 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.080009 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.080010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.080843 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.080844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.081534 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.081536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.081954 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.081955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.082012 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.082449 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.082454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.082842 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.084992 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.084992 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.084993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.085226 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.085423 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.085424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.085424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.085482 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.085755 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.085756 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.085787 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.086258 (+ 471us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.086343 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.254@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
I20260620 19:17:30.094776 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.087857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32884)
0620 19:17:30.087957 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.087988 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.088057 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.088181 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.088184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.088395 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.088434 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.088436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.089284 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.089285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.089825 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.089827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.090280 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.090281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.090331 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.090699 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.090703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.092118 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.093605 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.093611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.093613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.093864 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.094084 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.094084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.094085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.094150 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.094412 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.094418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.094468 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.094556 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.094663 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260620 19:17:30.094879 7650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.087910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.1.1.253:37587)
0620 19:17:30.087997 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.087999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.088043 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:17:30.088107 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.088108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.088116 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.088172 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.088505 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.089210 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.089901 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.090197 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.090393 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.092219 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.092220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.092220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.092226 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.093525 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.093950 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.093950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.093951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.093999 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.094238 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.094239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.094240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.094340 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.094403 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.094671 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.094771 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 19:17:30.095602 7114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.1.253:37587: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d"
member_type: NON_VOTER
last_known_addr {
host: "127.1.1.253"
port: 37587
}
attrs {
promote: true
}
}
dest_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af"
cas_config_opid_index: -1
I20260620 19:17:30.096032 7114 consensus_queue.cc:237] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } }
I20260620 19:17:30.096684 7673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8fbca32dc1904175b6babc7a337fca1d (127.1.1.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } }
I20260620 19:17:30.096954 7620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:30.097755 7673 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.1.253:37587
I20260620 19:17:30.097872 7620 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Refusing update from remote peer df95fcc3a9f6467e8cf4570c0d4d91af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:17:30.097981 7674 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } }
I20260620 19:17:30.098105 7578 master.cc:537] Master@127.1.1.253:37587 shutting down...
I20260620 19:17:30.098246 7673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:30.099094 7674 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
W20260620 19:17:30.100049 7101 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 19:17:30.100838 7578 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:30.100931 7578 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:30.100982 7578 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: stopping tablet replica
W20260620 19:17:30.101338 7101 consensus_peers.cc:487] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af -> Peer 8fbca32dc1904175b6babc7a337fca1d (127.1.1.253:37587): Couldn't send request to peer 8fbca32dc1904175b6babc7a337fca1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:17:30.113399 7578 master.cc:552] Master@127.1.1.253:37587 shutdown complete.
I20260620 19:17:30.113440 7578 master_runner.cc:305] Clearing existing system tablet
I20260620 19:17:30.113555 7578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.113597 7578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.114287 7578 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:30.114739 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.114863 7578 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.114918 7578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.114941 7578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "8fbca32dc1904175b6babc7a337fca1d"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "8c5bf52bdfbb4f28bf4bbc7ddfb97fbb"
server_key_iv: "0bd47f5e2fbd7985cc12f65707e3a56d"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.115574 7578 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:30.116886 7578 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:17:30.116951 7578 log.cc:1196] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 19:17:30.117174 7578 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Deleting consensus metadata
I20260620 19:17:30.117295 7578 master_runner.cc:315] Copying system tablet from 127.1.1.254:46261
I20260620 19:17:30.117795 7578 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: tablet copy: Beginning tablet copy session from remote peer at address 127.1.1.254:46261
I20260620 19:17:30.123376 7684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.118047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42208)
0620 19:17:30.118148 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.118157 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.118210 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.118327 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.118328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.118454 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.118507 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.118508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.119334 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.119335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.119884 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.119886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.120317 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.120317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.120359 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.120765 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.120768 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.121002 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.122594 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.122595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.122595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.122746 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.122932 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.122933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.122933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.122972 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.123171 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.123171 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.123212 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.123251 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.123319 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260620 19:17:30.123416 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.118068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42208 (local address 127.1.1.254:46261)
0620 19:17:30.118110 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.118111 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.118214 (+ 103us) server_negotiation.cc:407] Connection header received
0620 19:17:30.118241 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.118241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.118250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.118319 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.118565 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.119255 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.119954 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.120247 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.120314 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.121131 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.121133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.121133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.121139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.122522 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.122814 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.122816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.122816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.122871 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.123006 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.123007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.123007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.123118 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.123195 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.123273 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.123343 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:30.123629 7134 tablet_copy_service.cc:140] P df95fcc3a9f6467e8cf4570c0d4d91af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8fbca32dc1904175b6babc7a337fca1d ({username='oryx', principal='oryx/127.1.1.253@KRBTEST.COM'} at 127.0.0.1:42208)
I20260620 19:17:30.123704 7134 tablet_copy_service.cc:161] P df95fcc3a9f6467e8cf4570c0d4d91af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8fbca32dc1904175b6babc7a337fca1d at {username='oryx', principal='oryx/127.1.1.253@KRBTEST.COM'} at 127.0.0.1:42208: session id = 8fbca32dc1904175b6babc7a337fca1d-00000000000000000000000000000000
I20260620 19:17:30.124289 7134 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:17:30.124752 7578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:17:30.125710 7578 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: tablet copy: Starting download of 0 data blocks...
I20260620 19:17:30.125852 7578 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: tablet copy: Starting download of 1 WAL segments...
I20260620 19:17:30.126472 7578 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:17:30.127156 7134 tablet_copy_service.cc:342] P df95fcc3a9f6467e8cf4570c0d4d91af: Request end of tablet copy session 8fbca32dc1904175b6babc7a337fca1d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.1.253@KRBTEST.COM'} at 127.0.0.1:42208
I20260620 19:17:30.127197 7134 tablet_copy_service.cc:433] P df95fcc3a9f6467e8cf4570c0d4d91af: ending tablet copy session 8fbca32dc1904175b6babc7a337fca1d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8fbca32dc1904175b6babc7a337fca1d
I20260620 19:17:30.129498 7578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.130656 7686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.130719 7688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.130656 7578 server_base.cc:1034] running on GCE node
W20260620 19:17:30.130803 7685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.130892 7578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.131039 7578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.132145 7578 hybrid_clock.cc:648] HybridClock initialized: now 1781983050132139 us; error 20 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.133431 7578 init.cc:377] Logged in from keytab as oryx/127.1.1.253@KRBTEST.COM (short username oryx)
I20260620 19:17:30.133692 7578 webserver.cc:460] Webserver started at http://127.1.1.253:45667/ using document root <none> and password file <none>
I20260620 19:17:30.133788 7578 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.133832 7578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.134346 7578 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:30.134769 7695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.134914 7578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.134963 7578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "8fbca32dc1904175b6babc7a337fca1d"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "8c5bf52bdfbb4f28bf4bbc7ddfb97fbb"
server_key_iv: "0bd47f5e2fbd7985cc12f65707e3a56d"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.135010 7578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.158716 7578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.160501 7578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.163479 7578 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.253:37587
I20260620 19:17:30.163609 7747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.253:37587 every 8 connection(s)
I20260620 19:17:30.163854 7578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
I20260620 19:17:30.164530 7748 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:17:30.164916 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Bootstrap starting.
I20260620 19:17:30.167189 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:17:30.167407 7748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: Bootstrap complete.
I20260620 19:17:30.167593 7748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } }
I20260620 19:17:30.167668 7748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8fbca32dc1904175b6babc7a337fca1d, State: Initialized, Role: LEARNER
I20260620 19:17:30.167704 7748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } }
I20260620 19:17:30.167865 7748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:30.167850 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } }
I20260620 19:17:30.167953 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: This master's current role is: LEARNER
I20260620 19:17:30.169179 7762 catalog_manager.cc:1259] Loaded cluster ID: 945f86494ce8469e993f911b62ca54a7
I20260620 19:17:30.169222 7762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: loading cluster ID for follower catalog manager: success
I20260620 19:17:30.170111 7762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: acquiring CA information for follower catalog manager: success
I20260620 19:17:30.170353 7762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:17:30.178467 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.172732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.1.1.253:37587)
0620 19:17:30.172836 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.172838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.172846 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:30.172868 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.172869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.172882 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.172960 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.173521 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.174103 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.174754 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.174948 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.175026 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.176117 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.176118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.176118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.176125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.177420 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.177789 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.177790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.177790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.177842 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.178021 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.178022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.178022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.178118 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.178215 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.178309 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.178380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
WARNING: no policy specified for oryx/127.1.1.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.252.keytab.
Entry for principal oryx/127.1.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.252.keytab.
I20260620 19:17:30.204627 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:46167
--webserver_interface=127.1.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.252.keytab
--principal=oryx/127.1.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:17:30.287570 7766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:30.287742 7766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:30.287763 7766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:30.288780 7766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:17:30.288815 7766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:30.288828 7766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:30.288841 7766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:17:30.288853 7766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:17:30.290290 7766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.252.keytab
--principal=oryx/127.1.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:46167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:30.290462 7766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:30.290608 7766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.295094 7772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.295146 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.295099 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.295270 7766 server_base.cc:1034] running on GCE node
I20260620 19:17:30.295527 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.295828 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.296959 7766 hybrid_clock.cc:648] HybridClock initialized: now 1781983050296948 us; error 26 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.299577 7766 init.cc:377] Logged in from keytab as oryx/127.1.1.252@KRBTEST.COM (short username oryx)
I20260620 19:17:30.299917 7766 webserver.cc:460] Webserver started at http://127.1.1.252:41057/ using document root <none> and password file <none>
I20260620 19:17:30.300100 7766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.300139 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.300218 7766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:17:30.301219 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/instance:
uuid: "3fdec2ca49f94620aca477242ddb86db"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "5217f6c02abe152446421d87de65a5a5"
server_key_iv: "c223e523f4ea33a3d2d1370923b41c59"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.301507 7766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal/instance:
uuid: "3fdec2ca49f94620aca477242ddb86db"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "5217f6c02abe152446421d87de65a5a5"
server_key_iv: "c223e523f4ea33a3d2d1370923b41c59"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.302702 7766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:17:30.303347 7781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.303531 7766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:30.303604 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "3fdec2ca49f94620aca477242ddb86db"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "5217f6c02abe152446421d87de65a5a5"
server_key_iv: "c223e523f4ea33a3d2d1370923b41c59"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.303648 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.311277 7766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.312862 7766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:30.313014 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.317255 7766 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.252:46167
I20260620 19:17:30.317286 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.252:46167 every 8 connection(s)
I20260620 19:17:30.317566 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
I20260620 19:17:30.318363 7834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:30.318751 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7766
I20260620 19:17:30.319993 7834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.252@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
I20260620 19:17:30.328246 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.320818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.1.1.254:46261)
0620 19:17:30.320891 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.320893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.320900 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:30.320921 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.320921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.320930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.320980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.321402 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.322079 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.323281 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.323581 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.323630 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.325489 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.325491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.325491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.325497 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.326917 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.327439 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.327440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.327440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.327483 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.327722 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.327723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.327723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.327803 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.327844 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.328092 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.328157 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:17:30.328372 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.320625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42222)
0620 19:17:30.320781 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.320815 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.320868 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.321000 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.321002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.321243 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.321326 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.321329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.322195 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.322196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.323216 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.323218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.323646 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.323646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.323706 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.324096 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.324104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.325402 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.327029 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.327036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.327037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.327369 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.327558 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.327559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.327559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.327661 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.327856 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.327860 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.327978 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.328048 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.328165 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260620 19:17:30.328840 7834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:17:30.329013 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.319274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55012 (local address 127.1.1.252:46167)
0620 19:17:30.319493 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.319501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.319533 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:17:30.319992 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.320000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.320053 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.320191 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.320919 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.322102 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.323109 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.323504 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.323604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.326057 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.326063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.326063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.326070 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.327858 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.328150 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.328153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.328153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.328201 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.328393 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.328394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.328394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.328560 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.328721 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.328854 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.328916 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.252@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
I20260620 19:17:30.334357 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.329059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32916)
0620 19:17:30.329193 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.329202 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.329247 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.329403 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.329404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.329491 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.329523 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.329524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.330126 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.330127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.330545 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.330546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.330878 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.330879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.330919 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.331203 (+ 284us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.331206 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.332128 (+ 922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.333674 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.333675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.333675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.333820 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.333973 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.333974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.333974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.334018 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.334181 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.334181 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.334208 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.334252 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.334307 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 19:17:30.334444 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.329141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.1.1.253:37587)
0620 19:17:30.329177 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.329179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.329234 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:17:30.329300 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.329300 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.329309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.329395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.329604 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.330067 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.330631 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.330811 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.330871 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.332203 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.332204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.332205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.332211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.333587 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.333880 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.333881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.333881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.333923 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.334063 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.334063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.334063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.334138 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.334176 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.334299 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.334374 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:30.334800 7834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46167 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.252@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
I20260620 19:17:30.339839 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.334994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.0.0.1:55018)
0620 19:17:30.335033 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.335044 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.335077 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.335170 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.335170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.335260 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.335280 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.335281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.335855 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.335855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.336327 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.336329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.336621 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.336622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.336654 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.336929 (+ 275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.336932 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.337830 (+ 898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.339090 (+ 1260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.339090 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.339091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.339237 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.339429 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.339430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.339430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.339474 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.339632 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.339632 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.339660 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.339705 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.339768 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:17:30.339895 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.335047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.1.1.252:46167)
0620 19:17:30.335075 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.335076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.335083 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:30.335123 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.335123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.335129 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.335169 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.335349 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.335762 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.336393 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.336562 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.336611 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.337895 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.337896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.337896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.337901 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.339030 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.339295 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.339295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.339296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.339376 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.339510 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.339511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.339511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.339587 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.339629 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.339719 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.339809 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":13}
I20260620 19:17:30.340911 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Bootstrap starting.
I20260620 19:17:30.341605 7834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:30.341833 7834 log.cc:824] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:30.342469 7834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: No bootstrap required, opened a new log
I20260620 19:17:30.343710 7834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46167 } }
I20260620 19:17:30.343832 7834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:30.343871 7834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdec2ca49f94620aca477242ddb86db, State: Initialized, Role: FOLLOWER
I20260620 19:17:30.343976 7834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46167 } }
I20260620 19:17:30.344188 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46167 } } }
I20260620 19:17:30.344224 7834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:30.344274 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:17:30.345592 7850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:17:30.345661 7850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:17:30.347378 7766 master_runner.cc:428] Detected that this master 3fdec2ca49f94620aca477242ddb86db is joining an existing cluster
I20260620 19:17:30.347421 7766 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.1.252:46167
I20260620 19:17:30.354619 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.0.0.1:55030)
0620 19:17:30.348810 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.348820 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.349279 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.349398 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.349398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.349554 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.349601 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.349603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350328 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.350328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351012 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.351015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351459 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351506 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.351884 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.351888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.352217 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.353653 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.353654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.353654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.353818 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.353991 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.353992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.353992 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.354040 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.354294 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.354294 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.354335 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.354401 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.354495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260620 19:17:30.354660 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.1.1.252:46167)
0620 19:17:30.348540 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.348542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.349261 (+ 719us) server_negotiation.cc:407] Connection header received
0620 19:17:30.349330 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.349331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.349337 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.349358 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.349668 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350248 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351101 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351364 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351396 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.352307 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.352308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.352308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.352314 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.353563 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.353882 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.353884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.353884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.353933 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.354112 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.354112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.354113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.354229 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.354300 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.354431 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.354541 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":27}
I20260620 19:17:30.372923 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.1.1.253:37587)
0620 19:17:30.348379 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.348381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.348602 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:17:30.348652 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.348654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.348664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.348713 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.349560 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350157 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351215 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351477 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351509 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.370280 (+ 18771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.370281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.370281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.370287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.371678 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372168 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.372170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.372170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.372230 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372428 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.372429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.372430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.372538 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.372612 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.372764 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.372837 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:17:30.372973 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32928)
0620 19:17:30.348560 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.348567 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.348609 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.348752 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.348753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.349443 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.349489 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.349490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350242 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.350243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351126 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.351127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351568 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351627 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.369856 (+ 18229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.369860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.370223 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.371785 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.371786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.371786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.372072 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.372298 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.372299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.372347 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.372605 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.372606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.372657 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.372741 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.372838 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":72,"thread_start_us":66,"threads_started":1}
I20260620 19:17:30.373112 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.1.1.254:46261)
0620 19:17:30.348796 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.348798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.348804 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:30.348833 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.348833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.348851 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.348886 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.349547 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350167 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351210 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351464 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351501 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.370236 (+ 18735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.370238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.370238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.370245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.371655 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372318 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.372320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.372320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.372369 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372566 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.372567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.372567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.372682 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.372722 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.372916 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.372988 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260620 19:17:30.373402 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.348249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42232)
0620 19:17:30.348434 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.348443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.348659 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.348913 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.348914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.349393 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.349453 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.349454 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.350242 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.350243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351104 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.351106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.351568 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.351569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.351868 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.369668 (+ 17800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.369674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.370161 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.371750 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.371751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.371752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.372072 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.372439 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.372440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.372440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.372496 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.372767 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.372768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.372808 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.373202 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.373295 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":305,"thread_start_us":88,"threads_started":1}
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983049, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.254@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
I20260620 19:17:30.380295 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7176
I20260620 19:17:30.385812 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:40291
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:46261
--webserver_port=37563
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:17:30.386147 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.375149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.0.0.1:55032)
0620 19:17:30.375198 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.375208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.375264 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.375362 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.375363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.375499 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.375534 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.375535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.375993 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.375994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.376726 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.376729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.377058 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.377059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.377142 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.377442 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.377446 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.378590 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.380193 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.380194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.380194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.380399 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.385532 (+ 5133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.385533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.385534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.385596 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.385847 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.385848 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.385888 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.385956 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.386054 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:17:30.386201 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.375179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.1.1.252:46167)
0620 19:17:30.375210 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.375211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.375251 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:17:30.375316 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.375317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.375323 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.375358 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.375587 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.375937 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.376821 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.376980 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.377035 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.378677 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.378678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.378678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.378684 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.380051 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.385383 (+ 5332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.385384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.385384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.385452 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.385658 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.385659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.385659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.385766 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.385822 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.385992 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.386068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:17:30.387444 7114 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.1.252:46167: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3fdec2ca49f94620aca477242ddb86db"
member_type: NON_VOTER
last_known_addr {
host: "127.1.1.252"
port: 46167
}
attrs {
promote: true
}
}
dest_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af"
cas_config_opid_index: 5
I20260620 19:17:30.387657 7114 consensus_queue.cc:237] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:30.388031 7673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3fdec2ca49f94620aca477242ddb86db (127.1.1.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.389173 7673 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.1.252:46167
I20260620 19:17:30.389251 7673 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.389321 7673 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:30.389499 7766 master.cc:537] Master@127.1.1.252:46167 shutting down...
W20260620 19:17:30.390404 7098 consensus_peers.cc:487] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af -> Peer 3fdec2ca49f94620aca477242ddb86db (127.1.1.252:46167): Couldn't send request to peer 3fdec2ca49f94620aca477242ddb86db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 19:17:30.393052 7766 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:30.393187 7766 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:30.393239 7766 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: stopping tablet replica
I20260620 19:17:30.394598 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.388335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.0.0.1:32942)
0620 19:17:30.388368 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.388379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.388421 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.391312 (+ 2891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.391313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.391315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.391348 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.391349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.392249 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.392250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.393561 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.393565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.394321 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.394323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.394403 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.394426 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.394494 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.394537 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 19:17:30.394673 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.388211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.1.1.253:37587)
0620 19:17:30.391048 (+ 2837us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.391050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.391059 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:17:30.391081 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.391081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.391104 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.391167 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.391420 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.392158 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.393654 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.394229 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.394303 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.394361 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.394573 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.394610 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2816}
I20260620 19:17:30.395110 7722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 LEARNER]: Refusing update from remote peer df95fcc3a9f6467e8cf4570c0d4d91af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:17:30.395812 7673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:17:30.396344 7722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3fdec2ca49f94620aca477242ddb86db (127.1.1.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.397455 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df95fcc3a9f6467e8cf4570c0d4d91af. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.397550 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: This master's current role is: LEARNER
I20260620 19:17:30.397595 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: NON_VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: true } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.397647 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: This master's current role is: LEARNER
I20260620 19:17:30.398010 7860 raft_consensus.cc:1050] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: attempting to promote NON_VOTER 8fbca32dc1904175b6babc7a337fca1d to VOTER
I20260620 19:17:30.398185 7860 consensus_queue.cc:237] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:30.398602 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.398487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.0.0.1:55034)
0620 19:17:30.398517 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.398573 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 19:17:30.398813 7722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 LEARNER]: Refusing update from remote peer df95fcc3a9f6467e8cf4570c0d4d91af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 19:17:30.399009 7860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:17:30.399111 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.398826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.0.0.1:55046)
0620 19:17:30.399059 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.399087 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
W20260620 19:17:30.399490 7098 consensus_peers.cc:487] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af -> Peer 3fdec2ca49f94620aca477242ddb86db (127.1.1.252:46167): Couldn't send request to peer 3fdec2ca49f94620aca477242ddb86db. Status: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46167: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:17:30.399534 7673 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fbca32dc1904175b6babc7a337fca1d (127.1.1.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.400369 7673 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.400444 7673 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:30.400657 7674 sys_catalog.cc:455] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.400740 7674 sys_catalog.cc:458] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af [sys.catalog]: This master's current role is: LEADER
I20260620 19:17:30.400774 7722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8fbca32dc1904175b6babc7a337fca1d (127.1.1.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.401721 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.401850 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:17:30.407022 7766 master.cc:552] Master@127.1.1.252:46167 shutdown complete.
I20260620 19:17:30.407100 7766 master_runner.cc:305] Clearing existing system tablet
I20260620 19:17:30.407250 7766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.407303 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.407863 7766 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:17:30.408365 7865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.408499 7766 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.408550 7766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.408572 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "3fdec2ca49f94620aca477242ddb86db"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "5217f6c02abe152446421d87de65a5a5"
server_key_iv: "c223e523f4ea33a3d2d1370923b41c59"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.409332 7766 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:30.410568 7766 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:17:30.410631 7766 log.cc:1196] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 19:17:30.410806 7766 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Deleting consensus metadata
I20260620 19:17:30.410907 7766 master_runner.cc:315] Copying system tablet from 127.1.1.254:46261
I20260620 19:17:30.411394 7766 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: tablet copy: Beginning tablet copy session from remote peer at address 127.1.1.254:46261
I20260620 19:17:30.417992 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.411653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.1.1.254:46261)
0620 19:17:30.411693 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.411694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.411795 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:17:30.411841 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.411842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.411850 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.411913 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.412222 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.412949 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.413965 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.414256 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.414327 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.415494 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.415496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.415496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.415503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.416899 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.417274 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.417275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.417276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.417328 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.417516 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.417517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.417517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.417641 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.417713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.417854 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.417925 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:17:30.418001 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.411598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42236)
0620 19:17:30.411733 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.411741 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.411789 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.411918 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.411919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.412112 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.412151 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.412153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.413029 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.413030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.413825 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.413827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.414335 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.414336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.414398 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.415038 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.415043 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.415405 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.416970 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.416971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.416972 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.417202 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.417393 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.417393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.417394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.417451 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.417713 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.417714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.417753 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.417815 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.417900 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":51,"threads_started":1}
I20260620 19:17:30.418367 7134 tablet_copy_service.cc:140] P df95fcc3a9f6467e8cf4570c0d4d91af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3fdec2ca49f94620aca477242ddb86db ({username='oryx', principal='oryx/127.1.1.252@KRBTEST.COM'} at 127.0.0.1:42236)
I20260620 19:17:30.418414 7134 tablet_copy_service.cc:161] P df95fcc3a9f6467e8cf4570c0d4d91af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3fdec2ca49f94620aca477242ddb86db at {username='oryx', principal='oryx/127.1.1.252@KRBTEST.COM'} at 127.0.0.1:42236: session id = 3fdec2ca49f94620aca477242ddb86db-00000000000000000000000000000000
I20260620 19:17:30.418882 7134 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:17:30.419291 7766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:17:30.420490 7766 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: tablet copy: Starting download of 0 data blocks...
I20260620 19:17:30.420710 7766 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: tablet copy: Starting download of 1 WAL segments...
I20260620 19:17:30.421553 7766 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:17:30.422379 7134 tablet_copy_service.cc:342] P df95fcc3a9f6467e8cf4570c0d4d91af: Request end of tablet copy session 3fdec2ca49f94620aca477242ddb86db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.1.252@KRBTEST.COM'} at 127.0.0.1:42236
I20260620 19:17:30.422427 7134 tablet_copy_service.cc:433] P df95fcc3a9f6467e8cf4570c0d4d91af: ending tablet copy session 3fdec2ca49f94620aca477242ddb86db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3fdec2ca49f94620aca477242ddb86db
I20260620 19:17:30.424973 7766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.426174 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.429894 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.430078 7766 server_base.cc:1034] running on GCE node
W20260620 19:17:30.430123 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.430292 7766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.430408 7766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.431543 7766 hybrid_clock.cc:648] HybridClock initialized: now 1781983050431534 us; error 21 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.432925 7766 init.cc:377] Logged in from keytab as oryx/127.1.1.252@KRBTEST.COM (short username oryx)
I20260620 19:17:30.433223 7766 webserver.cc:460] Webserver started at http://127.1.1.252:43581/ using document root <none> and password file <none>
I20260620 19:17:30.433332 7766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.433357 7766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.433959 7766 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:30.434435 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.434572 7766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.434625 7766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "3fdec2ca49f94620aca477242ddb86db"
format_stamp: "Formatted at 2026-06-20 19:17:30 on dist-test-slave-mwkb"
server_key: "5217f6c02abe152446421d87de65a5a5"
server_key_iv: "c223e523f4ea33a3d2d1370923b41c59"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.434667 7766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.452836 7766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.454420 7766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.457603 7766 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.252:46167
I20260620 19:17:30.457693 7934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.252:46167 every 8 connection(s)
I20260620 19:17:30.458364 7766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
I20260620 19:17:30.459851 7935 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:17:30.460438 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Bootstrap starting.
I20260620 19:17:30.463327 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:17:30.463536 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: Bootstrap complete.
I20260620 19:17:30.463744 7935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:30.463840 7935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3fdec2ca49f94620aca477242ddb86db, State: Initialized, Role: LEARNER
I20260620 19:17:30.463876 7935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:30.464071 7935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:17:30.464075 7938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } } }
I20260620 19:17:30.464295 7938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db [sys.catalog]: This master's current role is: LEARNER
I20260620 19:17:30.465867 7949 catalog_manager.cc:1259] Loaded cluster ID: 945f86494ce8469e993f911b62ca54a7
I20260620 19:17:30.465972 7949 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: loading cluster ID for follower catalog manager: success
I20260620 19:17:30.467042 7949 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: acquiring CA information for follower catalog manager: success
I20260620 19:17:30.467339 7949 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fdec2ca49f94620aca477242ddb86db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:17:30.477787 7859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:30.477982 7859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:30.478005 7859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:30.479118 7859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:30.479156 7859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:30.479179 7859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:17:30.480614 7859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:40291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37563
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:30.480829 7859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:30.481021 7859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.486105 7954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.486124 7955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.486280 7859 server_base.cc:1034] running on GCE node
W20260620 19:17:30.486385 7957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.486590 7859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.486830 7859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.487998 7859 hybrid_clock.cc:648] HybridClock initialized: now 1781983050487988 us; error 31 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.490630 7859 init.cc:377] Logged in from keytab as oryx/127.1.1.193@KRBTEST.COM (short username oryx)
I20260620 19:17:30.491227 7859 webserver.cc:460] Webserver started at http://127.1.1.193:37563/ using document root <none> and password file <none>
I20260620 19:17:30.491487 7859 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.491555 7859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.493144 7859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:17:30.493863 7964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.494078 7859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:30.494166 7859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "583e924f8d654681f50597ed1b9106ad"
server_key_iv: "e0a204235aeebda3f291e86d9c7b7b3a"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.494544 7859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.517273 7859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.518847 7859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:30.519018 7859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.519232 7859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:30.519496 7859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:30.519527 7859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.519549 7859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:30.519562 7859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.525244 7859 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:40291
I20260620 19:17:30.525332 8077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:40291 every 8 connection(s)
I20260620 19:17:30.525660 7859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:17:30.530815 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 7859
I20260620 19:17:30.530977 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7310
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:30.536706 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.1.1.193:49297)
0620 19:17:30.526708 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.526718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.527501 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.527777 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.527777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.528090 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.528186 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.528188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.529287 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.529288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.529866 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.529868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530251 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.530369 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.530830 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.530838 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.533356 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.535114 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.535121 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.535123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.535410 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.535629 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.535629 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.535630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.535755 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.536085 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.536092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.536229 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.536324 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.536487 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260620 19:17:30.537165 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:49297 (local address 127.1.1.252:46167)
0620 19:17:30.526866 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.526867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.526874 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:30.527646 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.527647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.527658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.527727 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.528278 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.529216 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.529970 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530175 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530247 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.533460 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.533461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.533461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.533467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.535031 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.535488 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.535489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.535489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.535560 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.535827 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.535828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.535829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.535982 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.536046 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.536958 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.537041 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
I20260620 19:17:30.538365 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.1.1.193:42081)
0620 19:17:30.526477 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.526515 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.527482 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.527730 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.527734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.528005 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.528179 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.528183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.528825 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.528826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.529869 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.529871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530352 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.530403 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.531103 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.531213 (+ 110us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.535168 (+ 3955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.537079 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.537081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.537450 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.537670 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.537671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.537726 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.537997 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.537998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.538048 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.538114 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.538214 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260620 19:17:30.538654 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:42081 (local address 127.1.1.253:37587)
0620 19:17:30.526443 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.526445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.526603 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:17:30.527588 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.527589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.527599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.527656 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.528265 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.528725 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.529966 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530256 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530290 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.535319 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.535320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.535320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.535327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.537001 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537541 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.537542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.537543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.537601 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537784 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.537785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.537785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.537925 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.537970 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.538428 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.538518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:17:30.538946 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:36545 (local address 127.1.1.254:46261)
0620 19:17:30.526371 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.526373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.526633 (+ 260us) server_negotiation.cc:407] Connection header received
0620 19:17:30.527646 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.527647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.527658 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.527719 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.528277 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.528982 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.529959 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530228 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530285 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.534638 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.534640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.534640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.534658 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.536706 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537458 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.537459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.537459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.537516 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537720 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.537721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.537721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.537827 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.537895 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.538723 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.538807 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:17:30.539089 8080 heartbeater.cc:347] Connected to a master server at 127.1.1.253:37587
I20260620 19:17:30.539175 8080 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.539381 8080 heartbeater.cc:511] Master 127.1.1.253:37587 requested a full tablet report, sending...
I20260620 19:17:30.539498 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.526325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:36545)
0620 19:17:30.526554 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.526563 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.527501 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.527774 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.527776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.528093 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.528186 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.528188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.529061 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.529061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.529871 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.529872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.530333 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.530334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.530395 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.531134 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.531138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.534507 (+ 3369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.537006 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.537007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.537267 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.537584 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.537585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.537586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.537640 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.537901 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.537901 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.537939 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.538649 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.539139 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":35,"thread_start_us":83,"threads_started":1}
I20260620 19:17:30.539647 8078 heartbeater.cc:347] Connected to a master server at 127.1.1.252:46167
I20260620 19:17:30.539686 8078 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.539737 8078 heartbeater.cc:511] Master 127.1.1.252:46167 requested a full tablet report, sending...
I20260620 19:17:30.539863 7712 ts_manager.cc:194] Registered new tserver with Master: 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:30.541169 8079 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:30.541224 8079 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.541292 8079 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:30.541355 7899 ts_manager.cc:194] Registered new tserver with Master: 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:30.541551 7114 ts_manager.cc:194] Re-registered known tserver with Master: 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:30.541651 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:42555
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:46261
--webserver_port=41905
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:17:30.542008 7114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:36545
W20260620 19:17:30.620314 8086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:30.620496 8086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:30.620524 8086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:30.621693 8086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:30.621730 8086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:30.621806 8086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:17:30.623203 8086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:42555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=41905
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:30.623428 8086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:30.623632 8086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.628348 8094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.628345 8092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.628342 8091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.629268 8086 server_base.cc:1034] running on GCE node
I20260620 19:17:30.629528 8086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.629791 8086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.630934 8086 hybrid_clock.cc:648] HybridClock initialized: now 1781983050630912 us; error 38 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.634162 8086 init.cc:377] Logged in from keytab as oryx/127.1.1.194@KRBTEST.COM (short username oryx)
I20260620 19:17:30.634666 8086 webserver.cc:460] Webserver started at http://127.1.1.194:41905/ using document root <none> and password file <none>
I20260620 19:17:30.634896 8086 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.634970 8086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.636662 8086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:17:30.637316 8101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.637470 8086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.637566 8086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "21d9e09fb42c407d91228904eeef7dff"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "b46280f3a73a9667e5f82d61a52b2721"
server_key_iv: "76d49a0349cb727843f1d151a9b52ae4"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.637856 8086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.650413 8086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.652510 8086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:30.652657 8086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.652858 8086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:30.653180 8086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:30.653231 8086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.653280 8086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:30.653306 8086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.659166 8086 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:42555
I20260620 19:17:30.659210 8214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:42555 every 8 connection(s)
I20260620 19:17:30.659523 8086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:30.666405 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8086
I20260620 19:17:30.666560 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7444
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
I20260620 19:17:30.671636 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:36083 (local address 127.1.1.254:46261)
0620 19:17:30.660303 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.660305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.660580 (+ 275us) server_negotiation.cc:407] Connection header received
0620 19:17:30.661099 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.661101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.661113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.661187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.661878 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662770 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.663873 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664152 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664222 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.668201 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.668203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.668203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.668210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.669988 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670521 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.670522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.670523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.670578 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670834 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.670835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.670835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.670981 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.671023 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.671403 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.671487 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:17:30.671723 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:41783 (local address 127.1.1.253:37587)
0620 19:17:30.660231 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.660233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.660985 (+ 752us) server_negotiation.cc:407] Connection header received
0620 19:17:30.661165 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.661166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.661177 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.661223 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.661904 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662770 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.663826 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664123 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664174 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.668194 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.668196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.668196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.668203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.670437 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670900 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.670902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.670902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.670956 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.671162 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.671162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.671163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.671283 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.671323 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.671568 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.671634 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:17:30.671723 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:39511 (local address 127.1.1.252:46167)
0620 19:17:30.660707 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.660709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.660794 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:17:30.661125 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.661126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.661140 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.661173 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.661911 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662406 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.663705 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664012 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664080 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.668627 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.668628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.668629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.668635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.670552 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670958 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.670959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.670960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.671007 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.671211 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.671212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.671212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.671323 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.671363 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.671556 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.671635 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:17:30.672271 8221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.1.1.194:41783)
0620 19:17:30.660822 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.660891 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.660991 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.661262 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.661262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.661703 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.661782 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.661787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662866 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.662867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.663747 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.663748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664277 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.664334 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.665161 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.665165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.668086 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.670538 (+ 2452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.670540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.670818 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.671030 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.671031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.671031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.671089 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.671371 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.671372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.671430 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.671792 (+ 362us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.672037 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":676,"thread_start_us":68,"threads_started":1}
I20260620 19:17:30.672276 8219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:36083)
0620 19:17:30.660366 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.660398 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.660975 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.661256 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.661260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.661597 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.661780 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.661784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662869 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.662869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.663625 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.663627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664234 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.664319 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.664987 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.664992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.668087 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.670075 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.670083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.670432 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.670644 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.670645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.670768 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.671070 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.671076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.671228 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.671758 (+ 530us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.671956 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":62,"thread_start_us":101,"threads_started":1}
I20260620 19:17:30.672761 8216 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:30.672766 8217 heartbeater.cc:347] Connected to a master server at 127.1.1.253:37587
I20260620 19:17:30.672884 8217 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.672884 8216 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.673108 8217 heartbeater.cc:511] Master 127.1.1.253:37587 requested a full tablet report, sending...
I20260620 19:17:30.673525 7712 ts_manager.cc:194] Registered new tserver with Master: 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555)
I20260620 19:17:30.673110 8216 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:30.673712 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:41655
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:46261
--webserver_port=41805
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:17:30.673848 7114 ts_manager.cc:194] Re-registered known tserver with Master: 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555)
I20260620 19:17:30.674165 7114 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:36083
I20260620 19:17:30.674453 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.660597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.1.1.194:39511)
0620 19:17:30.660720 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.660730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.661006 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.661257 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.661260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.661641 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.661782 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.661787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.662503 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.662504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.663624 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.663626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.664095 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.664096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.664228 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.664830 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.664840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.668539 (+ 3699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.670627 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.670628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.670628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.670882 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.671089 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.671090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.671090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.671141 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.671407 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.671407 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.671438 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.673857 (+ 2419us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.673961 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260620 19:17:30.674770 8215 heartbeater.cc:347] Connected to a master server at 127.1.1.252:46167
I20260620 19:17:30.674826 8215 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.674891 8215 heartbeater.cc:511] Master 127.1.1.252:46167 requested a full tablet report, sending...
I20260620 19:17:30.675205 7899 ts_manager.cc:194] Registered new tserver with Master: 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555)
W20260620 19:17:30.751792 8222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:17:30.751966 8222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:17:30.751987 8222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:17:30.753046 8222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:17:30.753124 8222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:17:30.753160 8222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:17:30.754531 8222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:41655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=41805
--tserver_master_addrs=127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:17:30.754695 8222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:17:30.754851 8222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:17:30.759650 8227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.759742 8230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:17:30.759784 8228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:17:30.760052 8222 server_base.cc:1034] running on GCE node
I20260620 19:17:30.760303 8222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:17:30.760605 8222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:17:30.761799 8222 hybrid_clock.cc:648] HybridClock initialized: now 1781983050761773 us; error 57 us; skew 500 ppm
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:17:30.764439 8222 init.cc:377] Logged in from keytab as oryx/127.1.1.195@KRBTEST.COM (short username oryx)
I20260620 19:17:30.764932 8222 webserver.cc:460] Webserver started at http://127.1.1.195:41805/ using document root <none> and password file <none>
I20260620 19:17:30.765260 8222 fs_manager.cc:359] Metadata directory not provided
I20260620 19:17:30.765340 8222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:17:30.766723 8222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:30.767293 8237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.767462 8222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:17:30.767557 8222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "e6342558c60a40249311417c9ee394bb"
format_stamp: "Formatted at 2026-06-20 19:17:29 on dist-test-slave-mwkb"
server_key: "6414a78b4915c6fef1f133fcebaebe47"
server_key_iv: "3f54e40f13d360bfe117a56208bbb59a"
server_key_version: "encryptionkey@0"
I20260620 19:17:30.767871 8222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:17:30.799517 8222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:17:30.801661 8222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:17:30.801806 8222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:17:30.802021 8222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:17:30.802325 8222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:17:30.802377 8222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.802417 8222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:17:30.802446 8222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:17:30.807751 8222 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:41655
I20260620 19:17:30.807789 8350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:41655 every 8 connection(s)
I20260620 19:17:30.808125 8222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:17:30.808738 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8222
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.252@KRBTEST.COM
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.253@KRBTEST.COM
I20260620 19:17:30.818964 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.810064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42240 (local address 127.1.1.254:46261)
0620 19:17:30.810197 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.810199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.810205 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:17:30.810234 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.810234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.810243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.810320 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.811502 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.812322 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.813426 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.813724 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.814042 (+ 318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.815306 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.815308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.815308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.815315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.817576 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.817960 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.817961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.817961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.818015 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.818212 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.818213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.818213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.818339 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.818715 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.818774 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.818851 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
Jun 20 19:17:30 dist-test-slave-mwkb krb5kdc[7069](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983050, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:17:30.823125 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.809201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:52713 (local address 127.1.1.254:46261)
0620 19:17:30.809237 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.809239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.809334 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:17:30.810682 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.810682 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.810692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.810736 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.811631 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.812377 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.815391 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.815676 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.815733 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.820056 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.820058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.820058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.820064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.821925 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822378 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.822379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.822379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.822429 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822625 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.822625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.822626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.822732 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.822784 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.822985 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.823054 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:17:30.820890 7114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:17:30.823334 7114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:17:30.823443 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.808862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:33335 (local address 127.1.1.253:37587)
0620 19:17:30.808947 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.808948 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.809167 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:17:30.810685 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.810687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.810697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.810740 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.812634 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.813371 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.816114 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.816410 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.816467 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.818948 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.818949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.818950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.818956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.821491 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.821977 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.821979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.821979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.822030 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822336 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.822337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.822337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.822459 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.822520 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.823262 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.823340 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:17:30.823781 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.809044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:52713)
0620 19:17:30.809285 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.809293 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.810564 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.810843 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.810844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.811270 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.811512 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.811517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.814299 (+ 2782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.814299 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.815294 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.815296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.816720 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.816721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.816774 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.818253 (+ 1479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.818257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.819953 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.822068 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.822070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.822297 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.822492 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.822493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.822559 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.822807 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.822808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.822848 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.822900 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.823599 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":612,"thread_start_us":76,"threads_started":1}
I20260620 19:17:30.823894 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.808789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:37587 (local address 127.1.1.195:33335)
0620 19:17:30.809102 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.809113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.810552 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.810843 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.810844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.812471 (+ 1627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.812558 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.812559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.813455 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.813456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.816028 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.816031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.816512 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.816513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.816576 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.816946 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.816950 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.818850 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.821577 (+ 2727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.821583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.821584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.821893 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.822096 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.822127 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.822127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.822258 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.822546 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.822552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.822704 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.822819 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.822985 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":81,"threads_started":1}
I20260620 19:17:30.824223 8353 heartbeater.cc:347] Connected to a master server at 127.1.1.253:37587
I20260620 19:17:30.824312 8353 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.824373 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.808817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:34621 (local address 127.1.1.252:46167)
0620 19:17:30.808912 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.808914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.809143 (+ 229us) server_negotiation.cc:407] Connection header received
0620 19:17:30.810656 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.810658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.810670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.810700 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.811634 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.812368 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.813871 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.814191 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.814229 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.819475 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.819476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.819477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.819484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.822713 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.823446 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.823447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.823448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.823503 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.823723 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.823724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.823724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.823913 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.823977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.824216 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.824289 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 19:17:30.824477 8353 heartbeater.cc:511] Master 127.1.1.253:37587 requested a full tablet report, sending...
I20260620 19:17:30.824601 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.808788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46167 (local address 127.1.1.195:34621)
0620 19:17:30.809001 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.809037 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.810541 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.810840 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.810844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.811317 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:17:30.811526 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.811527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.812493 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.812494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.813780 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.813782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.814307 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.814308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.814429 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.814903 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:17:30.814911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:17:30.819338 (+ 4427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:17:30.823134 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.823135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.823136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.823353 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.823587 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:17:30.823588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:17:30.823588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:17:30.823645 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:17:30.824005 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:17:30.824006 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:17:30.824045 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.824107 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.824202 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260620 19:17:30.824625 8352 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46261
I20260620 19:17:30.824672 8352 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.824738 8352 heartbeater.cc:511] Master 127.1.1.254:46261 requested a full tablet report, sending...
I20260620 19:17:30.824934 7712 ts_manager.cc:194] Registered new tserver with Master: e6342558c60a40249311417c9ee394bb (127.1.1.195:41655)
I20260620 19:17:30.825001 7113 ts_manager.cc:194] Re-registered known tserver with Master: e6342558c60a40249311417c9ee394bb (127.1.1.195:41655)
I20260620 19:17:30.825030 8351 heartbeater.cc:347] Connected to a master server at 127.1.1.252:46167
I20260620 19:17:30.825084 8351 heartbeater.cc:464] Registering TS with master...
I20260620 19:17:30.825229 8351 heartbeater.cc:511] Master 127.1.1.252:46167 requested a full tablet report, sending...
I20260620 19:17:30.825474 7899 ts_manager.cc:194] Registered new tserver with Master: e6342558c60a40249311417c9ee394bb (127.1.1.195:41655)
I20260620 19:17:30.825522 7113 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:52713
I20260620 19:17:30.825593 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.809973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.1.1.252:46167)
0620 19:17:30.810063 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.810065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.814239 (+ 4174us) server_negotiation.cc:407] Connection header received
0620 19:17:30.814329 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.814330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.814342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.814376 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.815336 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.816021 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.817779 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.818088 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.818129 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.820390 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.820392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.820392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.820399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.823057 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.823705 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.823706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.823707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.823766 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.824812 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.824813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.824813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.824927 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.825005 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.825407 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.825483 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260620 19:17:30.826416 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.809981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32948 (local address 127.1.1.253:37587)
0620 19:17:30.810975 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.810990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.810998 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:30.811017 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.811018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.811026 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.811596 (+ 570us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.812263 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.813064 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.814140 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.815762 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.815833 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.817303 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.817304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.817305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.817311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.820763 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.825563 (+ 4800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.825564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.825565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.825615 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.825826 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.825827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.825827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.825946 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.826018 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.826200 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.826269 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":46,"threads_started":1}
I20260620 19:17:30.832149 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.828767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:42555 (local address 127.0.0.1:48214)
0620 19:17:30.828820 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.828831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.828867 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.829507 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.829507 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.829508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.829537 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.829538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.830287 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.830288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.830990 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.830992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.831935 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.831991 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.831995 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.832052 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.832089 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":35}
I20260620 19:17:30.832319 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.828874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48214 (local address 127.1.1.194:42555)
0620 19:17:30.828995 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.828998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.829006 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:30.829397 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.829400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.829419 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.829596 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.829647 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.830194 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831064 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.831840 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831934 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.831994 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.832064 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.832114 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260620 19:17:30.832458 8374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.828984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40291 (local address 127.0.0.1:42670)
0620 19:17:30.829246 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.829256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.829340 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.829556 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.829557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.829558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.829586 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.829587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.830265 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.830266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.831278 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.831280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.832254 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.832255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.832315 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.832319 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.832361 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.832400 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":21,"thread_start_us":56,"threads_started":1}
I20260620 19:17:30.832598 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.829093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.1.1.193:40291)
0620 19:17:30.829221 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.829224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.829315 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:17:30.829425 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.829427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.829442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.829501 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.829639 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.830190 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831391 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.831850 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831931 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.831995 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.832393 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.832438 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 19:17:30.833128 8012 tablet_service.cc:1467] Processing CreateTablet for tablet e37c94a0e93a44d483d9580246801d3a (DEFAULT_TABLE table=test-table [id=feba9d6fdfc54ca09c008a02cbcb20a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:30.833130 8149 tablet_service.cc:1467] Processing CreateTablet for tablet e37c94a0e93a44d483d9580246801d3a (DEFAULT_TABLE table=test-table [id=feba9d6fdfc54ca09c008a02cbcb20a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:30.833487 8012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37c94a0e93a44d483d9580246801d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:30.833487 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37c94a0e93a44d483d9580246801d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:30.833709 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.828846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:41655 (local address 127.0.0.1:35754)
0620 19:17:30.828942 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.828950 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.829086 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.830458 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.830459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.830461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.830496 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.830498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.831459 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.831460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.832470 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.832473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.833339 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.833340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.833405 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.833409 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.833456 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.833498 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260620 19:17:30.833967 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.829272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.1.1.195:41655)
0620 19:17:30.830334 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.830337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.830344 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:17:30.830370 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.830372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.830384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.830449 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.830564 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.831408 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.832576 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.833218 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.833325 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.833386 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.833865 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.833899 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":65,"threads_started":1}
I20260620 19:17:30.834491 8285 tablet_service.cc:1467] Processing CreateTablet for tablet e37c94a0e93a44d483d9580246801d3a (DEFAULT_TABLE table=test-table [id=feba9d6fdfc54ca09c008a02cbcb20a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:17:30.834774 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e37c94a0e93a44d483d9580246801d3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:17:30.836952 8377 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Bootstrap starting.
I20260620 19:17:30.837532 8377 tablet_bootstrap.cc:654] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:30.837589 8379 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Bootstrap starting.
I20260620 19:17:30.837810 8377 log.cc:824] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:30.838068 8378 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Bootstrap starting.
I20260620 19:17:30.838141 8379 tablet_bootstrap.cc:654] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:30.838405 8379 log.cc:824] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:30.838438 8377 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: No bootstrap required, opened a new log
I20260620 19:17:30.838485 8377 ts_tablet_manager.cc:1397] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:30.838528 8378 tablet_bootstrap.cc:654] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Neither blocks nor log segments found. Creating new log.
I20260620 19:17:30.838752 8378 log.cc:824] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Log is configured to *not* fsync() on all Append() calls
I20260620 19:17:30.839044 8379 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: No bootstrap required, opened a new log
I20260620 19:17:30.839112 8379 ts_tablet_manager.cc:1397] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:17:30.839426 8378 tablet_bootstrap.cc:492] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: No bootstrap required, opened a new log
I20260620 19:17:30.839494 8378 ts_tablet_manager.cc:1397] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:17:30.840695 8377 raft_consensus.cc:348] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.840873 8377 raft_consensus.cc:374] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:30.840907 8377 raft_consensus.cc:729] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49a26f476dbc4a2d9c2abf8d33f7cfc1, State: Initialized, Role: FOLLOWER
I20260620 19:17:30.841034 8377 consensus_queue.cc:260] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.841300 8377 ts_tablet_manager.cc:1428] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:17:30.841331 8079 heartbeater.cc:503] Master 127.1.1.254:46261 was elected leader, sending a full tablet report...
I20260620 19:17:30.841547 8379 raft_consensus.cc:348] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.841701 8379 raft_consensus.cc:374] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:30.841737 8379 raft_consensus.cc:729] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6342558c60a40249311417c9ee394bb, State: Initialized, Role: FOLLOWER
I20260620 19:17:30.841833 8379 consensus_queue.cc:260] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.842036 8379 ts_tablet_manager.cc:1428] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:17:30.842131 8352 heartbeater.cc:503] Master 127.1.1.254:46261 was elected leader, sending a full tablet report...
I20260620 19:17:30.842545 8378 raft_consensus.cc:348] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.842691 8378 raft_consensus.cc:374] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:17:30.842736 8378 raft_consensus.cc:729] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d9e09fb42c407d91228904eeef7dff, State: Initialized, Role: FOLLOWER
I20260620 19:17:30.842864 8378 consensus_queue.cc:260] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.843103 8378 ts_tablet_manager.cc:1428] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260620 19:17:30.843458 8216 heartbeater.cc:503] Master 127.1.1.254:46261 was elected leader, sending a full tablet report...
I20260620 19:17:30.882627 8384 raft_consensus.cc:482] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:17:30.882748 8384 raft_consensus.cc:504] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.883054 8384 leader_election.cc:288] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555), 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:30.886085 8373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.883316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:38195 (local address 127.1.1.194:42555)
0620 19:17:30.883387 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.883389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.883399 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:17:30.883424 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.883425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.883437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.883501 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.883676 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.884203 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885202 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.885755 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885813 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.885822 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.885980 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.886012 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:17:30.886096 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.883520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:37713 (local address 127.1.1.193:40291)
0620 19:17:30.883569 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.883571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.883579 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:17:30.883600 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.883601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.883612 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.883666 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:17:30.883806 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.884256 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885210 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.885692 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.885793 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.885996 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.886030 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:17:30.886094 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.883241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:42555 (local address 127.1.1.195:38195)
0620 19:17:30.883303 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.883316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.883369 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.883512 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.883513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.883515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.883590 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.883593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.884279 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.884280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.885065 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.885131 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.885822 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.885906 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.885909 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.885975 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.886021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:17:30.886106 8355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.883387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40291 (local address 127.1.1.195:37713)
0620 19:17:30.883437 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:30.883446 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:17:30.883495 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:17:30.883688 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:17:30.883689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:17:30.883691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:17:30.883743 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.883744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.884349 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.884350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.885062 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:17:30.885109 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.885825 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.885826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:17:30.885912 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.885915 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:17:30.885979 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:17:30.886023 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260620 19:17:30.886525 8032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37c94a0e93a44d483d9580246801d3a" candidate_uuid: "e6342558c60a40249311417c9ee394bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" is_pre_election: true
I20260620 19:17:30.886620 8169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37c94a0e93a44d483d9580246801d3a" candidate_uuid: "e6342558c60a40249311417c9ee394bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d9e09fb42c407d91228904eeef7dff" is_pre_election: true
I20260620 19:17:30.886682 8032 raft_consensus.cc:2454] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6342558c60a40249311417c9ee394bb in term 0.
I20260620 19:17:30.886778 8169 raft_consensus.cc:2454] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6342558c60a40249311417c9ee394bb in term 0.
I20260620 19:17:30.886888 8241 leader_election.cc:302] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a26f476dbc4a2d9c2abf8d33f7cfc1, e6342558c60a40249311417c9ee394bb; no voters:
I20260620 19:17:30.887017 8384 raft_consensus.cc:2781] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:17:30.887094 8384 raft_consensus.cc:482] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:17:30.887130 8384 raft_consensus.cc:3037] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:30.887811 8384 raft_consensus.cc:504] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.887955 8384 leader_election.cc:288] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [CANDIDATE]: Term 1 election: Requested vote from peers 21d9e09fb42c407d91228904eeef7dff (127.1.1.194:42555), 49a26f476dbc4a2d9c2abf8d33f7cfc1 (127.1.1.193:40291)
I20260620 19:17:30.888301 8032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37c94a0e93a44d483d9580246801d3a" candidate_uuid: "e6342558c60a40249311417c9ee394bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1"
I20260620 19:17:30.888339 8169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e37c94a0e93a44d483d9580246801d3a" candidate_uuid: "e6342558c60a40249311417c9ee394bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21d9e09fb42c407d91228904eeef7dff"
I20260620 19:17:30.888396 8169 raft_consensus.cc:3037] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:30.888388 8032 raft_consensus.cc:3037] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:17:30.889101 8032 raft_consensus.cc:2454] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6342558c60a40249311417c9ee394bb in term 1.
I20260620 19:17:30.889227 8169 raft_consensus.cc:2454] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6342558c60a40249311417c9ee394bb in term 1.
I20260620 19:17:30.889267 8241 leader_election.cc:302] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49a26f476dbc4a2d9c2abf8d33f7cfc1, e6342558c60a40249311417c9ee394bb; no voters:
I20260620 19:17:30.889338 8384 raft_consensus.cc:2781] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:17:30.889505 8384 raft_consensus.cc:686] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 1 LEADER]: Becoming Leader. State: Replica: e6342558c60a40249311417c9ee394bb, State: Running, Role: LEADER
I20260620 19:17:30.889616 8384 consensus_queue.cc:237] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } }
I20260620 19:17:30.890374 7114 catalog_manager.cc:5495] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb reported cstate change: term changed from 0 to 1, leader changed from <none> to e6342558c60a40249311417c9ee394bb (127.1.1.195). New cstate: current_term: 1 leader_uuid: "e6342558c60a40249311417c9ee394bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6342558c60a40249311417c9ee394bb" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 41655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 } health_report { overall_health: UNKNOWN } } }
I20260620 19:17:30.905164 8376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.902815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.1.1.195:41655)
0620 19:17:30.902875 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.902877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.902887 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:17:30.902945 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.902946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.902964 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.903059 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:17:30.903195 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.903648 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.904351 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.904569 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.904653 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.904822 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:17:30.904926 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:17:30.905008 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.905052 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.905089 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:17:30.906728 8169 raft_consensus.cc:1261] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 1 FOLLOWER]: Refusing update from remote peer e6342558c60a40249311417c9ee394bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:30.906728 8032 raft_consensus.cc:1261] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 1 FOLLOWER]: Refusing update from remote peer e6342558c60a40249311417c9ee394bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:17:30.907012 8384 consensus_queue.cc:1035] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "21d9e09fb42c407d91228904eeef7dff" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:30.907130 8386 consensus_queue.cc:1035] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "49a26f476dbc4a2d9c2abf8d33f7cfc1" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:17:30.910002 8390 mvcc.cc:205] Tried to move back new op lower bound from 7299002576511692800 to 7299002576444473344. Current Snapshot: MvccSnapshot[applied={T|T < 7299002576511692800}]
I20260620 19:17:30.910111 8389 mvcc.cc:205] Tried to move back new op lower bound from 7299002576511692800 to 7299002576444473344. Current Snapshot: MvccSnapshot[applied={T|T < 7299002576511692800 or (T in {7299002576511692800})}]
I20260620 19:17:30.910856 8391 mvcc.cc:205] Tried to move back new op lower bound from 7299002576511692800 to 7299002576444473344. Current Snapshot: MvccSnapshot[applied={T|T < 7299002576511692800}]
I20260620 19:17:30.911176 7113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:30.911263 7113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:17:30.912451 7113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df95fcc3a9f6467e8cf4570c0d4d91af: Sending DeleteTablet for 3 replicas of tablet e37c94a0e93a44d483d9580246801d3a
I20260620 19:17:30.912786 8012 tablet_service.cc:1514] Processing DeleteTablet for tablet e37c94a0e93a44d483d9580246801d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:30 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:42670
I20260620 19:17:30.912791 8285 tablet_service.cc:1514] Processing DeleteTablet for tablet e37c94a0e93a44d483d9580246801d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:30 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:35754
I20260620 19:17:30.913151 8399 tablet_replica.cc:330] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: stopping tablet replica
I20260620 19:17:30.913168 8149 tablet_service.cc:1514] Processing DeleteTablet for tablet e37c94a0e93a44d483d9580246801d3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:17:30 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:48214
I20260620 19:17:30.913236 8399 raft_consensus.cc:2229] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:17:30.913336 8399 raft_consensus.cc:2258] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:30.913388 8400 tablet_replica.cc:330] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: stopping tablet replica
I20260620 19:17:30.913475 8400 raft_consensus.cc:2229] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:30.913571 8400 raft_consensus.cc:2258] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:30.913756 8399 ts_tablet_manager.cc:1905] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:30.914006 8400 ts_tablet_manager.cc:1905] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:30.914170 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7082
I20260620 19:17:30.915424 8398 tablet_replica.cc:330] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: stopping tablet replica
I20260620 19:17:30.915553 8398 raft_consensus.cc:2229] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:17:30.915696 8398 raft_consensus.cc:2258] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:17:30.916501 8398 ts_tablet_manager.cc:1905] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:17:30.917671 8400 ts_tablet_manager.cc:1918] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:30.917740 8400 log.cc:1196] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/e37c94a0e93a44d483d9580246801d3a
I20260620 19:17:30.917994 8400 ts_tablet_manager.cc:1939] T e37c94a0e93a44d483d9580246801d3a P 21d9e09fb42c407d91228904eeef7dff: Deleting consensus metadata
I20260620 19:17:30.918205 8398 ts_tablet_manager.cc:1918] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:30.918314 8398 log.cc:1196] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/e37c94a0e93a44d483d9580246801d3a
I20260620 19:17:30.918610 8398 ts_tablet_manager.cc:1939] T e37c94a0e93a44d483d9580246801d3a P 49a26f476dbc4a2d9c2abf8d33f7cfc1: Deleting consensus metadata
W20260620 19:17:30.919564 7968 connection.cc:535] server connection from 127.0.0.1:42670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:17:30.920346 8105 connection.cc:535] server connection from 127.0.0.1:48214 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:17:30.921509 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.913754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.1.1.252:46167)
0620 19:17:30.913788 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.913789 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.913923 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:17:30.914009 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.914010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.914022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.914049 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.915371 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.916043 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.916892 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.917180 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.917220 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.918665 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.918667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.918667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.918673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.920328 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.920722 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.920723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.920724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.920774 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.920944 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.920945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.920945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.921061 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.921143 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.921337 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.921418 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:17:30.922324 8399 ts_tablet_manager.cc:1918] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:17:30.922399 8399 log.cc:1196] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/e37c94a0e93a44d483d9580246801d3a
I20260620 19:17:30.922689 8399 ts_tablet_manager.cc:1939] T e37c94a0e93a44d483d9580246801d3a P e6342558c60a40249311417c9ee394bb: Deleting consensus metadata
W20260620 19:17:30.922996 8241 connection.cc:441] server connection from 127.0.0.1:35754 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35754 (request call id 1) could send its response
I20260620 19:17:30.928402 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.922121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32964 (local address 127.1.1.253:37587)
0620 19:17:30.922170 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.922172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.922350 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:17:30.922451 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.922452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.922465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.922495 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.923260 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.923695 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.924338 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.924539 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.924616 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.925991 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.925993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.925993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.925999 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.927344 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.927646 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.927647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.927647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.927698 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.927886 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.927887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.927887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.928008 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.928121 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.928243 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.928314 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:17:30.930001 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:30.922657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.1.1.252:46167)
0620 19:17:30.922688 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:17:30.922690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:17:30.923010 (+ 320us) server_negotiation.cc:407] Connection header received
0620 19:17:30.923150 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:17:30.923151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:17:30.923160 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:17:30.923189 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:17:30.923849 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.924492 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.925637 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:17:30.925938 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:17:30.926098 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:17:30.927339 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:17:30.927340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:17:30.927341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:17:30.927346 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:17:30.928897 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.929338 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.929339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.929339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.929396 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:17:30.929575 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:17:30.929576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:17:30.929576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:17:30.929691 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:17:30.929778 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:17:30.929836 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:17:30.929913 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:17:31.911227 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:31.910981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:41343)
0620 19:17:31.911106 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:31.911177 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260620 19:17:31.911227 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:31.910981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:53567)
0620 19:17:31.911105 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:31.911177 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260620 19:17:31.911446 8352 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
W20260620 19:17:31.911458 8216 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:31.911649 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:31.911019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:60493)
0620 19:17:31.911380 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:31.911612 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":50,"threads_started":1}
W20260620 19:17:31.911890 8079 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:46261 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:32.412763 8412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:32.412886 8412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:32.413193 8412 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:32.413511 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:32.413302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42724)
0620 19:17:32.413424 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:32.413469 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
W20260620 19:17:32.413633 7696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:32.413959 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:32.413862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42730)
0620 19:17:32.413914 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:32.413938 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:17:32.414057 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:32.414096 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:32.414177 8412 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:32.912072 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:32.911838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:57199)
0620 19:17:32.911985 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:32.912020 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260620 19:17:32.912139 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:32.911821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:36289)
0620 19:17:32.911966 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:32.912004 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260620 19:17:32.912513 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:32.912236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:60503)
0620 19:17:32.912349 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:32.912465 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
I20260620 19:17:33.912797 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:33.912556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:43709)
0620 19:17:33.912700 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:33.912738 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260620 19:17:33.913134 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:33.912870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:52541)
0620 19:17:33.913029 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:33.913092 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":58,"threads_started":1}
I20260620 19:17:33.913192 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:33.912797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:42623)
0620 19:17:33.913110 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:33.913146 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":67,"threads_started":1}
I20260620 19:17:34.035985 8421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:34.036064 8421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:34.036238 8421 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:34.036513 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:34.036351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42036)
0620 19:17:34.036440 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:34.036473 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260620 19:17:34.036885 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:34.036803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42052)
0620 19:17:34.036849 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:34.036870 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 19:17:34.036954 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:34.037005 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:34.037101 8421 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:34.913386 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:34.913177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:35535)
0620 19:17:34.913293 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:34.913334 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":63,"threads_started":1}
I20260620 19:17:34.913877 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:34.913584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:41723)
0620 19:17:34.913691 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:34.913729 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":43,"threads_started":1}
I20260620 19:17:34.914623 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:34.914410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:41993)
0620 19:17:34.914555 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:34.914583 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":42,"threads_started":1}
I20260620 19:17:35.914156 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:35.913919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:43697)
0620 19:17:35.914048 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:35.914101 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260620 19:17:35.914618 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:35.914417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:37107)
0620 19:17:35.914539 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:35.914576 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":45,"threads_started":1}
I20260620 19:17:35.915184 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:35.914970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:57027)
0620 19:17:35.915117 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:35.915144 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":42,"threads_started":1}
I20260620 19:17:36.571508 8430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:36.571628 8430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:36.571811 8430 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:36.572188 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:36.572008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42312)
0620 19:17:36.572111 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:36.572152 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260620 19:17:36.572576 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:36.572483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42320)
0620 19:17:36.572532 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:36.572555 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 19:17:36.572655 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:36.572702 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:36.572791 8430 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:36.914799 8433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:36.914524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:52969)
0620 19:17:36.914698 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:36.914740 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260620 19:17:36.915108 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:36.914931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:50107)
0620 19:17:36.915038 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:36.915071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
I20260620 19:17:36.915836 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:36.915664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:41117)
0620 19:17:36.915774 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:36.915804 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260620 19:17:37.915657 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:37.915487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:40597)
0620 19:17:37.915590 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:37.915617 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
I20260620 19:17:37.915705 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:37.915424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:37811)
0620 19:17:37.915568 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:37.915602 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260620 19:17:37.916326 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:37.916147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:55809)
0620 19:17:37.916248 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:37.916285 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":65,"threads_started":1}
I20260620 19:17:38.916332 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:38.916111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:60029)
0620 19:17:38.916243 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:38.916278 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260620 19:17:38.916626 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:38.916160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:44639)
0620 19:17:38.916413 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:38.916578 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":52,"threads_started":1}
I20260620 19:17:38.916847 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:38.916605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:57747)
0620 19:17:38.916778 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:38.916810 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":48,"threads_started":1}
I20260620 19:17:39.917302 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:39.917057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:51903)
0620 19:17:39.917203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:39.917243 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260620 19:17:39.917606 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:39.917397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:55415)
0620 19:17:39.917539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:39.917566 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":43,"threads_started":1}
I20260620 19:17:39.917574 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:39.917254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:38899)
0620 19:17:39.917423 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:39.917459 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":46,"threads_started":1}
I20260620 19:17:40.918239 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:40.918018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:59757)
0620 19:17:40.918166 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:40.918195 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260620 19:17:40.918310 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:40.918018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:59279)
0620 19:17:40.918146 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:40.918187 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260620 19:17:40.918422 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:40.918046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:56811)
0620 19:17:40.918338 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:40.918377 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":55,"threads_started":1}
I20260620 19:17:40.997231 8448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:40.997417 8448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:40.997723 8448 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:40.998577 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:40.998087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42650)
0620 19:17:40.998466 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:40.998543 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":283,"threads_started":1}
W20260620 19:17:40.998664 7696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260620 19:17:40.999123 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:40.998980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:42658)
0620 19:17:40.999050 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:40.999099 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:17:40.999341 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:40.999428 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:40.999526 8448 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:41.919026 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:41.918746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:49501)
0620 19:17:41.918913 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:41.918956 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260620 19:17:41.919090 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:41.918904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:49739)
0620 19:17:41.919004 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:41.919040 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260620 19:17:41.919275 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:41.919086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:53679)
0620 19:17:41.919201 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:41.919235 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":42,"threads_started":1}
I20260620 19:17:42.919741 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:42.919514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:35359)
0620 19:17:42.919640 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:42.919679 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260620 19:17:42.919829 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:42.919542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:49597)
0620 19:17:42.919727 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:42.919760 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260620 19:17:42.919972 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:42.919751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:36601)
0620 19:17:42.919900 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:42.919930 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":49,"threads_started":1}
I20260620 19:17:43.871964 8457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:43.872058 8457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:43.872236 8457 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:43.872588 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:43.872409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:48014)
0620 19:17:43.872508 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:43.872552 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":55,"threads_started":1}
I20260620 19:17:43.872980 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:43.872880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:48016)
0620 19:17:43.872929 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:43.872962 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:17:43.873054 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:43.873132 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:43.873215 8457 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:43.920483 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:43.920192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:52843)
0620 19:17:43.920389 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:43.920427 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260620 19:17:43.920557 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:43.920244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:52261)
0620 19:17:43.920466 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:43.920504 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":41,"threads_started":1}
I20260620 19:17:43.920670 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:43.920471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:40791)
0620 19:17:43.920597 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:43.920627 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260620 19:17:44.921204 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:44.920979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:33405)
0620 19:17:44.921112 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:44.921151 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 19:17:44.921236 8464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:44.921009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:45485)
0620 19:17:44.921151 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:44.921186 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260620 19:17:44.921430 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:44.921019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:50585)
0620 19:17:44.921354 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:44.921389 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":66,"threads_started":1}
I20260620 19:17:45.922082 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:45.921848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:34825)
0620 19:17:45.921989 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:45.922029 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260620 19:17:45.922084 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:45.921895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:60941)
0620 19:17:45.922027 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:45.922051 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260620 19:17:45.922231 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:45.921913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:42463)
0620 19:17:45.922152 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:45.922187 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":56,"threads_started":1}
I20260620 19:17:46.922788 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:46.922532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:45029)
0620 19:17:46.922670 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:46.922715 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260620 19:17:46.922832 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:46.922599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:41649)
0620 19:17:46.922743 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:46.922780 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 19:17:46.922967 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:46.922760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:58887)
0620 19:17:46.922895 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:46.922926 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":49,"threads_started":1}
I20260620 19:17:47.923498 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:47.923308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:41091)
0620 19:17:47.923421 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:47.923464 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":39,"threads_started":1}
I20260620 19:17:47.923481 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:47.923249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:56421)
0620 19:17:47.923361 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:47.923409 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260620 19:17:47.923509 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:47.923215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:57875)
0620 19:17:47.923403 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:47.923463 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":63,"threads_started":1}
I20260620 19:17:48.924096 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:48.923902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:59947)
0620 19:17:48.924005 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:48.924039 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260620 19:17:48.924110 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:48.923900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:36409)
0620 19:17:48.924021 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:48.924056 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260620 19:17:48.924290 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:48.924131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:36983)
0620 19:17:48.924222 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:48.924253 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":56,"threads_started":1}
I20260620 19:17:49.924778 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:49.924560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:39569)
0620 19:17:49.924683 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:49.924725 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260620 19:17:49.924857 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:49.924548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:38359)
0620 19:17:49.924772 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:49.924810 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":61,"threads_started":1}
I20260620 19:17:49.924861 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:49.924683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:39845)
0620 19:17:49.924802 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:49.924828 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":44,"threads_started":1}
I20260620 19:17:50.925437 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:50.925231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:59213)
0620 19:17:50.925371 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:50.925400 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":53,"threads_started":1}
I20260620 19:17:50.925437 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:50.925230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:45105)
0620 19:17:50.925348 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:50.925386 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260620 19:17:50.925523 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:50.925240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:49727)
0620 19:17:50.925437 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:50.925476 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":46,"threads_started":1}
I20260620 19:17:51.078544 8484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:17:51.078666 8484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:17:51.078850 8484 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:17:51.079239 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:51.079019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:48362)
0620 19:17:51.079162 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:51.079202 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260620 19:17:51.079320 7696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260620 19:17:51.079653 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:51.079561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:48374)
0620 19:17:51.079613 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:51.079638 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:17:51.079738 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:17:51.079789 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:17:51.079869 8484 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:17:51.926059 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:51.925877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:57473)
0620 19:17:51.925963 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:51.926002 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":41,"threads_started":1}
I20260620 19:17:51.926229 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:51.925838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:45291)
0620 19:17:51.926157 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:51.926191 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":60,"threads_started":1}
I20260620 19:17:51.926277 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:51.926047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:53963)
0620 19:17:51.926184 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:51.926230 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":52,"threads_started":1}
I20260620 19:17:52.926862 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:52.926649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:44783)
0620 19:17:52.926760 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:52.926797 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260620 19:17:52.926916 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:52.926691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:50025)
0620 19:17:52.926836 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:52.926871 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":42,"threads_started":1}
I20260620 19:17:52.926916 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:52.926622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:41855)
0620 19:17:52.926857 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:52.926882 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":72,"threads_started":1}
I20260620 19:17:53.927511 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:53.927281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:42975)
0620 19:17:53.927406 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:53.927450 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 19:17:53.927546 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:53.927281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:51011)
0620 19:17:53.927461 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:53.927495 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":63,"threads_started":1}
I20260620 19:17:53.927608 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:53.927445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:48239)
0620 19:17:53.927532 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:53.927565 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
I20260620 19:17:54.928226 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:54.928009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:50881)
0620 19:17:54.928127 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:54.928170 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":46,"threads_started":1}
I20260620 19:17:54.928311 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:54.927971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:56587)
0620 19:17:54.928223 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:54.928257 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":62,"threads_started":1}
I20260620 19:17:54.928493 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:54.928003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:47547)
0620 19:17:54.928344 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:54.928449 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":56,"threads_started":1}
I20260620 19:17:55.928974 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:55.928685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:60771)
0620 19:17:55.928883 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:55.928919 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":57,"threads_started":1}
I20260620 19:17:55.929023 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:55.928758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:41425)
0620 19:17:55.928958 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:55.928988 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":47,"threads_started":1}
I20260620 19:17:55.929337 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:55.929057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:49359)
0620 19:17:55.929263 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:55.929295 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":57,"threads_started":1}
I20260620 19:17:56.929941 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:56.929705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:46669)
0620 19:17:56.929852 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:56.929890 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260620 19:17:56.929941 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:56.929751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:50553)
0620 19:17:56.929852 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:56.929890 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":46,"threads_started":1}
I20260620 19:17:56.929944 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:56.929574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:34385)
0620 19:17:56.929855 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:56.929891 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":71,"threads_started":1}
I20260620 19:17:57.930585 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:57.930337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:59529)
0620 19:17:57.930456 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:57.930493 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
I20260620 19:17:57.930648 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:57.930432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:40359)
0620 19:17:57.930565 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:57.930600 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":56,"threads_started":1}
I20260620 19:17:57.930667 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:57.930432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:52695)
0620 19:17:57.930598 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:57.930631 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":53,"threads_started":1}
I20260620 19:17:58.931298 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:58.930968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:49565)
0620 19:17:58.931196 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:58.931237 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":59,"threads_started":1}
I20260620 19:17:58.931367 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:58.931052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:50865)
0620 19:17:58.931286 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:58.931323 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":58,"threads_started":1}
I20260620 19:17:58.931367 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:58.930968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:51245)
0620 19:17:58.931297 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:58.931325 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":59,"threads_started":1}
I20260620 19:17:59.932032 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:59.931796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:58341)
0620 19:17:59.931933 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:59.931974 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260620 19:17:59.932159 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:59.931795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:56451)
0620 19:17:59.932074 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:59.932112 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":93,"threads_started":1}
I20260620 19:17:59.932191 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:17:59.932009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:37113)
0620 19:17:59.932118 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:17:59.932149 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":45,"threads_started":1}
I20260620 19:18:00.531867 8514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df95fcc3a9f6467e8cf4570c0d4d91af)
I20260620 19:18:00.531950 8514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "df95fcc3a9f6467e8cf4570c0d4d91af" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46261 } } peers { permanent_uuid: "8fbca32dc1904175b6babc7a337fca1d" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 37587 } attrs { promote: false } } peers { permanent_uuid: "3fdec2ca49f94620aca477242ddb86db" member_type: NON_VOTER last_known_addr { host: "127.1.1.252" port: 46167 } attrs { promote: true } }
I20260620 19:18:00.532157 8514 leader_election.cc:288] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261)
I20260620 19:18:00.532492 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:00.532316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:54210)
0620 19:18:00.532416 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:00.532447 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":55,"threads_started":1}
W20260620 19:18:00.532582 7696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:18:00.532919 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:00.532835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.0.0.1:54212)
0620 19:18:00.532880 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:00.532904 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:18:00.532991 7696 leader_election.cc:334] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df95fcc3a9f6467e8cf4570c0d4d91af (127.1.1.254:46261): Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
I20260620 19:18:00.533099 7696 leader_election.cc:302] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 8fbca32dc1904175b6babc7a337fca1d; no voters: df95fcc3a9f6467e8cf4570c0d4d91af
I20260620 19:18:00.533217 8514 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8fbca32dc1904175b6babc7a337fca1d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:18:00.932822 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:00.932583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.194:53991)
0620 19:18:00.932728 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:00.932768 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260620 19:18:00.932875 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:00.932592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.195:43181)
0620 19:18:00.932745 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:00.932779 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260620 19:18:00.932926 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:00.932774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46261 (local address 127.1.1.193:49519)
0620 19:18:00.932861 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:00.932888 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:46261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":40,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.1.1.254:46261,127.1.1.253:37587,127.1.1.252:46167, num_attempts: 1)
I20260620 19:18:01.102339 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7859
I20260620 19:18:01.107091 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8086
I20260620 19:18:01.111397 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8222
I20260620 19:18:01.116374 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7578
I20260620 19:18:01.120906 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 7766
2026-06-20T19:18:01Z chronyd exiting
I20260620 19:18:01.136706 1031 test_util.cc:175] -----------------------------------------------
I20260620 19:18:01.136750 1031 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781983021660275-1031-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (35361 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:18:01 dist-test-slave-mwkb krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:18:01 dist-test-slave-mwkb krb5kdc[8522](info): set up 2 sockets
Jun 20 19:18:01 dist-test-slave-mwkb krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260620 19:18:02.097155 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab.
Entry for principal oryx/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab.
I20260620 19:18:02.203369 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35619
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab
--principal=oryx/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:35619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:02.277494 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:02.277722 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:02.277756 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:02.278795 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:02.278854 8535 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:18:02.278875 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:02.278896 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:02.278921 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:02.278944 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:02.280387 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:35619
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.254.keytab
--principal=oryx/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:02.280651 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:02.280885 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:18:02.285492 8535 server_base.cc:1034] running on GCE node
W20260620 19:18:02.285455 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.285598 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.285749 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.286027 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:02.286264 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:02.287418 8535 hybrid_clock.cc:648] HybridClock initialized: now 1781983082287411 us; error 36 us; skew 500 ppm
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:02.289654 8535 init.cc:377] Logged in from keytab as oryx/127.1.1.254@KRBTEST.COM (short username oryx)
I20260620 19:18:02.290061 8535 webserver.cc:460] Webserver started at http://127.1.1.254:44827/ using document root <none> and password file <none>
I20260620 19:18:02.290285 8535 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:02.290334 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:02.290441 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:02.291320 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "6152c0764c664a7a9a25939f88464d70"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "c468bfffbcdc6066ded32eaa6c954260"
server_key_iv: "d514654540fc199630b92e86dacda8da"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.291647 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "6152c0764c664a7a9a25939f88464d70"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "c468bfffbcdc6066ded32eaa6c954260"
server_key_iv: "d514654540fc199630b92e86dacda8da"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.292795 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:18:02.293485 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.293655 8535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.293736 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "6152c0764c664a7a9a25939f88464d70"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "c468bfffbcdc6066ded32eaa6c954260"
server_key_iv: "d514654540fc199630b92e86dacda8da"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.293805 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:02.312830 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:02.314363 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:02.314508 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:02.318930 8535 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:35619
I20260620 19:18:02.319037 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:35619 every 8 connection(s)
I20260620 19:18:02.319329 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:02.319907 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:02.321873 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Bootstrap starting.
I20260620 19:18:02.322366 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:02.322587 8613 log.cc:824] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:02.323117 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: No bootstrap required, opened a new log
I20260620 19:18:02.324855 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } }
I20260620 19:18:02.324983 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:02.325017 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6152c0764c664a7a9a25939f88464d70, State: Initialized, Role: FOLLOWER
I20260620 19:18:02.325177 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } }
I20260620 19:18:02.325240 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:02.325261 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:02.325289 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:02.325755 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } }
I20260620 19:18:02.325835 8613 leader_election.cc:302] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6152c0764c664a7a9a25939f88464d70; no voters:
I20260620 19:18:02.325949 8613 leader_election.cc:288] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:18:02.326015 8616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:02.326184 8616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [term 1 LEADER]: Becoming Leader. State: Replica: 6152c0764c664a7a9a25939f88464d70, State: Running, Role: LEADER
I20260620 19:18:02.326233 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:02.326269 8616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } }
I20260620 19:18:02.326642 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6152c0764c664a7a9a25939f88464d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } } }
I20260620 19:18:02.326671 8618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6152c0764c664a7a9a25939f88464d70. Latest consensus state: current_term: 1 leader_uuid: "6152c0764c664a7a9a25939f88464d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6152c0764c664a7a9a25939f88464d70" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35619 } } }
I20260620 19:18:02.326819 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:02.326838 8618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:02.327139 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8535
I20260620 19:18:02.327250 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:18:02.327412 8625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:02.327436 1031 external_mini_cluster.cc:1428] Setting key ee4295d596f64a4cf4f9048046bf684a
I20260620 19:18:02.327952 8625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:18:02.329707 8625 catalog_manager.cc:1347] Generated new cluster ID: 58dab0d6142543c1affb507b1498ad2a
I20260620 19:18:02.329758 8625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:02.337713 8625 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:18:02.338363 8625 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:02.338884 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.328017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.1.1.254:35619)
0620 19:18:02.328370 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.328380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.328409 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:18:02.328680 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.328690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.328763 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.328903 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.329742 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.331206 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.332225 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.332825 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.332933 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.335554 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.335557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.335558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.335568 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.337309 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.337758 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.337760 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.337761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.337870 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.338064 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.338065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.338065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.338277 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.338443 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.338558 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.338704 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":311,"thread_start_us":190,"threads_started":1}
I20260620 19:18:02.344154 8625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Generated new TSK 0
I20260620 19:18:02.344295 8625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
I20260620 19:18:02.375623 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35619
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:02.450007 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:02.450179 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:02.450199 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:02.451184 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:02.451215 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:02.451233 8639 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:18:02.451249 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:02.452507 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:02.452653 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:02.452800 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:02.457269 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.457275 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.457288 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.457402 8639 server_base.cc:1034] running on GCE node
I20260620 19:18:02.457617 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:02.457777 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:02.458913 8639 hybrid_clock.cc:648] HybridClock initialized: now 1781983082458902 us; error 24 us; skew 500 ppm
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:02.461303 8639 init.cc:377] Logged in from keytab as oryx/127.1.1.193@KRBTEST.COM (short username oryx)
I20260620 19:18:02.461723 8639 webserver.cc:460] Webserver started at http://127.1.1.193:41925/ using document root <none> and password file <none>
I20260620 19:18:02.461894 8639 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:02.461930 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:02.462018 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:02.462884 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "f4d86a23204d4b13a757d9a0de4bed00"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "02ddc91e2d3cd40619438506bcf9fd5b"
server_key_iv: "b2262098f467d26c378a5532c3e2f5e2"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.463152 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "f4d86a23204d4b13a757d9a0de4bed00"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "02ddc91e2d3cd40619438506bcf9fd5b"
server_key_iv: "b2262098f467d26c378a5532c3e2f5e2"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.464269 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:02.464915 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.465142 8639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:02.465235 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "f4d86a23204d4b13a757d9a0de4bed00"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "02ddc91e2d3cd40619438506bcf9fd5b"
server_key_iv: "b2262098f467d26c378a5532c3e2f5e2"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.465315 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:02.472162 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:02.474128 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:02.474268 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:02.474805 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:02.474862 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.474902 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:02.474931 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:02.481876 8639 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:36607
I20260620 19:18:02.481936 8769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:36607 every 8 connection(s)
I20260620 19:18:02.482245 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:18:02.483650 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.475152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:53445 (local address 127.1.1.254:35619)
0620 19:18:02.475211 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.475214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.475452 (+ 238us) server_negotiation.cc:407] Connection header received
0620 19:18:02.476129 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.476130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.476142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.476184 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.476994 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.477485 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.478358 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.478562 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.478630 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.480938 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.480940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.480940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.480947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.482615 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.482939 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.482940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.482940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.483018 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.483225 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.483225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.483225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.483321 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.483369 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.483536 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.483595 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:18:02.483783 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.475149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.193:53445)
0620 19:18:02.475309 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.475356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.476021 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.476250 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.476267 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.476622 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.476897 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.476903 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.477579 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.477580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.478259 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.478261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.478636 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.478637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.478742 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.479258 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.479267 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.480839 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.482683 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.482685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.482687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.482875 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.483085 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.483086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.483086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.483170 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.483360 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.483362 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.483461 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.483523 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.483624 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 19:18:02.484223 8770 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35619
I20260620 19:18:02.484297 8770 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:02.484449 8770 heartbeater.cc:511] Master 127.1.1.254:35619 requested a full tablet report, sending...
I20260620 19:18:02.484968 8566 ts_manager.cc:194] Registered new tserver with Master: f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607)
I20260620 19:18:02.485839 8566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:53445
I20260620 19:18:02.489491 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8639
I20260620 19:18:02.489603 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:18:02.489827 1031 external_mini_cluster.cc:1428] Setting key 28f7e3340716fe2c3369af2c96d3d771
I20260620 19:18:02.490355 8776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.485193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.193:45569)
0620 19:18:02.485310 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.485320 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.485376 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.485488 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.485489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.485634 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.485678 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.485679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.486384 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.486385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.486959 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.486961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.487240 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.487241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.487281 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.487643 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.487647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.487889 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.489218 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.489219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.489219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.489457 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.489700 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.489701 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.489701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.489765 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.490071 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.490073 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.490115 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.490187 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.490249 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260620 19:18:02.490362 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.485254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:45569 (local address 127.1.1.254:35619)
0620 19:18:02.485294 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.485296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.485371 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:18:02.485424 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.485425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.485434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.485486 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.485739 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.486280 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.487025 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.487177 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.487219 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.487950 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.487951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.487951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.487955 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.489157 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.489528 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.489528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.489528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.489599 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.489841 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.489842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.489843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.489973 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.490047 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.490221 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.490287 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
I20260620 19:18:02.511214 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35619
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:02.585213 8779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:02.585443 8779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:02.585479 8779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:02.586632 8779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:02.586692 8779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:02.586735 8779 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:18:02.586763 8779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:02.588104 8779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:02.588325 8779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:02.588516 8779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:02.593202 8784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.593242 8787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.593276 8779 server_base.cc:1034] running on GCE node
W20260620 19:18:02.593221 8785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.593612 8779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:02.593876 8779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:02.595017 8779 hybrid_clock.cc:648] HybridClock initialized: now 1781983082595018 us; error 39 us; skew 500 ppm
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:02.597263 8779 init.cc:377] Logged in from keytab as oryx/127.1.1.194@KRBTEST.COM (short username oryx)
I20260620 19:18:02.597728 8779 webserver.cc:460] Webserver started at http://127.1.1.194:43507/ using document root <none> and password file <none>
I20260620 19:18:02.597937 8779 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:02.597996 8779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:02.598116 8779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:02.599013 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "5a5fd376a5a1434ab06152e51243ba0c"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "45c894e3fbfbd433fdc45bacde9360ac"
server_key_iv: "65f4b88f3707b988ad596fdb7cd1555d"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.599359 8779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "5a5fd376a5a1434ab06152e51243ba0c"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "45c894e3fbfbd433fdc45bacde9360ac"
server_key_iv: "65f4b88f3707b988ad596fdb7cd1555d"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.600587 8779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:02.601359 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.601522 8779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:02.601617 8779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "5a5fd376a5a1434ab06152e51243ba0c"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "45c894e3fbfbd433fdc45bacde9360ac"
server_key_iv: "65f4b88f3707b988ad596fdb7cd1555d"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.601696 8779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:02.610329 8779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:02.611807 8779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:02.611943 8779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:02.612411 8779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:02.612473 8779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.612541 8779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:02.612573 8779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:02.619064 8779 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:34373
I20260620 19:18:02.619180 8909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:34373 every 8 connection(s)
I20260620 19:18:02.619433 8779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:18:02.621798 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.612908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:38863 (local address 127.1.1.254:35619)
0620 19:18:02.612983 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.612985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.613191 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:18:02.613616 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.613617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.613631 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.613699 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.614442 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.615327 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.616794 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.617137 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.617211 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.619474 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.619475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.619476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.619482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.620746 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.621108 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.621109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.621110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.621171 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.621338 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.621339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.621339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.621423 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.621467 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.621679 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.621739 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:18:02.621923 8803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.612883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.194:38863)
0620 19:18:02.613051 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.613105 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.613520 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.613754 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.613756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.614097 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.614344 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.614349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.615421 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.615423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.616706 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.616709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.617244 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.617245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.617400 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.617902 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.617910 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.619381 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.620811 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.620816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.620817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.621012 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.621217 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.621217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.621217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.621300 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.621473 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.621477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.621575 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.621649 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.621749 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260620 19:18:02.622347 8910 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35619
I20260620 19:18:02.622416 8910 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:02.622562 8910 heartbeater.cc:511] Master 127.1.1.254:35619 requested a full tablet report, sending...
I20260620 19:18:02.622893 8567 ts_manager.cc:194] Registered new tserver with Master: 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373)
I20260620 19:18:02.623275 8567 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:38863
I20260620 19:18:02.624954 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8779
I20260620 19:18:02.625087 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:18:02.625277 1031 external_mini_cluster.cc:1428] Setting key 6fe2bec9d1d1fe19d7ee7186f4b94a86
I20260620 19:18:02.629719 8916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.623497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.194:34249)
0620 19:18:02.623626 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.623634 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.623706 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.623812 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.623813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.623967 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.624007 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.624008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.624576 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.624577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.625374 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.625377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.625874 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.625875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.625944 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.626341 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.626345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.626592 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.628195 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.628196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.628197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.628474 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.628769 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.628770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.628770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.628840 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.629340 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.629341 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.629391 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.629470 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.629576 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260620 19:18:02.629806 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.623542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:34249 (local address 127.1.1.254:35619)
0620 19:18:02.623579 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.623581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.623695 (+ 114us) server_negotiation.cc:407] Connection header received
0620 19:18:02.623757 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.623758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.623766 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.623809 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.624089 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.624484 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.625481 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.625764 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.625870 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.626703 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.626705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.626705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.626711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.628096 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.628571 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.628572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.628573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.628691 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.628951 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.628952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.628952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.629241 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.629366 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.629499 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.629633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
I20260620 19:18:02.650029 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35619
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:02.723711 8919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:02.723884 8919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:02.723918 8919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:02.724962 8919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:02.725025 8919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:02.725060 8919 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:18:02.725118 8919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:02.726692 8919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.1.254:35619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:02.726972 8919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:02.727183 8919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:02.731695 8925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.731739 8919 server_base.cc:1034] running on GCE node
W20260620 19:18:02.731791 8927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:02.731940 8924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:02.732165 8919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:02.732393 8919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:02.733548 8919 hybrid_clock.cc:648] HybridClock initialized: now 1781983082733534 us; error 24 us; skew 500 ppm
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:02.735864 8919 init.cc:377] Logged in from keytab as oryx/127.1.1.195@KRBTEST.COM (short username oryx)
I20260620 19:18:02.736274 8919 webserver.cc:460] Webserver started at http://127.1.1.195:45941/ using document root <none> and password file <none>
I20260620 19:18:02.736447 8919 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:02.736493 8919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:02.736579 8919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:02.737511 8919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "2495de1a715346e48896bf0037eeddf6"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "eddc1926275d2b06516f5d8fe7d99a83"
server_key_iv: "e067686eee79cee8ba850a63d9917968"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.737795 8919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "2495de1a715346e48896bf0037eeddf6"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "eddc1926275d2b06516f5d8fe7d99a83"
server_key_iv: "e067686eee79cee8ba850a63d9917968"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.738931 8919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:02.739596 8934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.739779 8919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:02.739857 8919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "2495de1a715346e48896bf0037eeddf6"
format_stamp: "Formatted at 2026-06-20 19:18:02 on dist-test-slave-mwkb"
server_key: "eddc1926275d2b06516f5d8fe7d99a83"
server_key_iv: "e067686eee79cee8ba850a63d9917968"
server_key_version: "encryptionkey@0"
I20260620 19:18:02.739907 8919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:02.747442 8919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:02.748879 8919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:02.748978 8919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:02.749392 8919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:02.749430 8919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.749491 8919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:02.749521 8919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:02.755434 8919 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:32883
I20260620 19:18:02.755539 9049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:32883 every 8 connection(s)
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:02.755681 8919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:18:02.758316 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.749776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:50219 (local address 127.1.1.254:35619)
0620 19:18:02.749822 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.749825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.749978 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:18:02.750352 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.750353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.750366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.750442 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.751061 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.751809 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.753256 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.753571 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.753642 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.756187 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.756188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.756189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.756194 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.757380 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.757703 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.757704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.757704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.757746 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.757919 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.757919 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.757920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.757997 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.758037 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.758214 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.758263 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:18:02.758507 8943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.749756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.195:50219)
0620 19:18:02.749872 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.749903 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.750270 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.750482 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.750485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.750734 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.750965 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.750970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.751911 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.751912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.753158 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.753161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.753659 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.753660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.753785 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.754304 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.754315 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.756100 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.757441 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.757446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.757447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.757643 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.757797 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.757797 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.757797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.757875 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.758041 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.758044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.758131 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.758200 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.758294 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":63,"threads_started":1}
I20260620 19:18:02.758885 9050 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35619
I20260620 19:18:02.758970 9050 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:02.759174 9050 heartbeater.cc:511] Master 127.1.1.254:35619 requested a full tablet report, sending...
I20260620 19:18:02.759531 8566 ts_manager.cc:194] Registered new tserver with Master: 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:02.759985 8566 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:50219
I20260620 19:18:02.763793 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 8919
I20260620 19:18:02.763911 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:18:02.764091 1031 external_mini_cluster.cc:1428] Setting key c7f6330c0d77012c7b4577a5cdf3b0a9
I20260620 19:18:02.764928 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:18:02.764938 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.759901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:52669 (local address 127.1.1.254:35619)
0620 19:18:02.759937 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.759938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.760049 (+ 111us) server_negotiation.cc:407] Connection header received
0620 19:18:02.760322 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.760323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.760333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.760393 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.760680 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.761200 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.761803 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.761952 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.761993 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.762702 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.762703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.762703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.762707 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.763768 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.764178 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.764179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.764179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.764232 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.764426 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.764427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.764427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.764553 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.764623 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.764791 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.764858 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:18:02.764969 9056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.759853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.1.1.195:52669)
0620 19:18:02.759977 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.759986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.760068 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.760407 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.760408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.760561 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:02.760606 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.760607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.761275 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.761276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.761742 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.761743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.762004 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.762005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.762046 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.762409 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:02.762413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:02.762642 (+ 229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:02.763861 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.763862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.763862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.764083 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.764304 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:02.764305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:02.764306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:02.764363 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:02.764647 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:02.764648 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:02.764687 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.764768 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.764859 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:18:02.783632 8635 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:02.779149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.1.1.254:35619)
0620 19:18:02.779192 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.779194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.779202 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:02.779268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.779269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.779278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.779339 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.779872 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.780314 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.780964 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.781167 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.781227 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.783594 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53696: BlockingRecv error: recv got EOF from 127.0.0.1:53696 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 19:18:02 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:02.789825 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.784158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.1.1.254:35619)
0620 19:18:02.784195 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.784197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.784330 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:18:02.784389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.784389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.784396 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.784437 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:02.784845 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.785363 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.785968 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.786118 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.786160 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.787925 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:02.787926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:02.787926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:02.787931 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:02.788996 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.789295 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.789296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.789296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.789338 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:02.789476 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:02.789476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:02.789476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:02.789545 (+ 69us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:02.789602 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.789658 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.789731 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:18:02.791210 8566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:18:02.792666 8566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:02.798602 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.0.0.1:50050)
0620 19:18:02.794709 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.794746 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.794806 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.795064 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.795066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.795068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:02.795126 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.795127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796254 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.796255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.797349 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.797351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798321 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.798395 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798409 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.798481 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.798528 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":61,"threads_started":1}
I20260620 19:18:02.798629 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.1.1.193:36607)
0620 19:18:02.794867 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.794872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.794883 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:18:02.794921 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.794923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.794941 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.795047 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:02.795220 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796180 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.797572 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798245 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798310 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798371 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.798491 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.798528 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260620 19:18:02.798702 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:32883 (local address 127.0.0.1:39480)
0620 19:18:02.795145 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.795155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.795208 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.795369 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.795370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.795372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:02.795410 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.795411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796696 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.796696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.797734 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.797736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798510 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.798559 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798562 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.798603 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.798638 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":52,"threads_started":1}
I20260620 19:18:02.798906 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.1.1.195:32883)
0620 19:18:02.795026 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.795029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.795213 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:18:02.795261 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.795264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.795279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.795362 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:02.795504 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796609 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.797814 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798358 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798418 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798480 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.798774 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.798814 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 19:18:02.798910 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:34373 (local address 127.0.0.1:45468)
0620 19:18:02.794817 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:02.794826 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:02.794866 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:02.795586 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:02.795587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:02.795589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:02.795620 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.795621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796808 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.796809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.797835 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:02.797837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798700 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:02.798759 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798762 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:02.798813 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:18:02.798848 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":45,"threads_started":1}
I20260620 19:18:02.798967 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:02.794682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.1.1.194:34373)
0620 19:18:02.795309 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:02.795313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:02.795320 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:18:02.795370 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:02.795372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:02.795445 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:02.795518 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:02.795693 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.796719 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.797909 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:02.798531 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:02.798604 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:02.798664 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:02.798827 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:18:02.798859 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":66,"threads_started":1}
I20260620 19:18:02.799597 8844 tablet_service.cc:1467] Processing CreateTablet for tablet aafde880bf2548b786930d780ce164a8 (DEFAULT_TABLE table=test-table [id=24ab13b88c07455b9aba6832cee223eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:02.799750 8984 tablet_service.cc:1467] Processing CreateTablet for tablet aafde880bf2548b786930d780ce164a8 (DEFAULT_TABLE table=test-table [id=24ab13b88c07455b9aba6832cee223eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:02.799914 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafde880bf2548b786930d780ce164a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:02.799845 8704 tablet_service.cc:1467] Processing CreateTablet for tablet aafde880bf2548b786930d780ce164a8 (DEFAULT_TABLE table=test-table [id=24ab13b88c07455b9aba6832cee223eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:02.800077 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafde880bf2548b786930d780ce164a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:02.800144 8704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafde880bf2548b786930d780ce164a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:02.802585 9075 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Bootstrap starting.
I20260620 19:18:02.803234 9075 tablet_bootstrap.cc:654] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:02.803606 9077 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Bootstrap starting.
I20260620 19:18:02.803647 9075 log.cc:824] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:02.804229 9077 tablet_bootstrap.cc:654] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:02.804446 9075 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: No bootstrap required, opened a new log
I20260620 19:18:02.804525 9075 ts_tablet_manager.cc:1397] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:02.804541 9077 log.cc:824] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:02.804862 9076 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Bootstrap starting.
I20260620 19:18:02.805272 9077 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: No bootstrap required, opened a new log
I20260620 19:18:02.805351 9077 ts_tablet_manager.cc:1397] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:02.805465 9076 tablet_bootstrap.cc:654] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:02.806099 9076 log.cc:824] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:02.806641 9075 raft_consensus.cc:348] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.806797 9075 raft_consensus.cc:374] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:02.806809 9076 tablet_bootstrap.cc:492] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: No bootstrap required, opened a new log
I20260620 19:18:02.806835 9075 raft_consensus.cc:729] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Initialized, Role: FOLLOWER
I20260620 19:18:02.806859 9076 ts_tablet_manager.cc:1397] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:18:02.806972 9075 consensus_queue.cc:260] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.806926 9077 raft_consensus.cc:348] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.807071 9077 raft_consensus.cc:374] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:02.807111 9077 raft_consensus.cc:729] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5fd376a5a1434ab06152e51243ba0c, State: Initialized, Role: FOLLOWER
I20260620 19:18:02.807219 9075 ts_tablet_manager.cc:1428] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:18:02.807269 9050 heartbeater.cc:503] Master 127.1.1.254:35619 was elected leader, sending a full tablet report...
I20260620 19:18:02.807228 9077 consensus_queue.cc:260] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.807734 9077 ts_tablet_manager.cc:1428] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:18:02.807783 8910 heartbeater.cc:503] Master 127.1.1.254:35619 was elected leader, sending a full tablet report...
I20260620 19:18:02.808657 9076 raft_consensus.cc:348] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.808770 9076 raft_consensus.cc:374] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:02.808790 9076 raft_consensus.cc:729] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Initialized, Role: FOLLOWER
I20260620 19:18:02.808880 9076 consensus_queue.cc:260] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:02.809149 9076 ts_tablet_manager.cc:1428] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:18:02.809209 8770 heartbeater.cc:503] Master 127.1.1.254:35619 was elected leader, sending a full tablet report...
W20260620 19:18:02.870213 8911 tablet.cc:2369] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:18:02.982890 8771 tablet.cc:2369] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:18:03.006201 9051 tablet.cc:2369] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:18:03.233139 9083 raft_consensus.cc:482] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:03.233327 9083 raft_consensus.cc:504] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.233692 9083 leader_election.cc:288] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:03.236800 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.234030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:41191 (local address 127.1.1.195:32883)
0620 19:18:03.234087 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.234089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.234114 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:18:03.234171 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.234172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.234184 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.234239 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:03.234441 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.235144 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236150 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.236487 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236538 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.236545 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.236700 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.236738 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:18:03.236799 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.233913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:32883 (local address 127.1.1.193:41191)
0620 19:18:03.234053 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:03.234065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:03.234111 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:03.234279 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:03.234280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:03.234282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:03.234363 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.234366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.235213 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.235214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.236035 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.236037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.236564 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.236630 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.236635 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:18:03.236684 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:18:03.236713 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260620 19:18:03.236871 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.234009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:41737 (local address 127.1.1.194:34373)
0620 19:18:03.234054 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.234057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.234258 (+ 201us) server_negotiation.cc:407] Connection header received
0620 19:18:03.234286 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.234287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.234300 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.234376 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:03.234569 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.235278 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236214 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.236542 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236607 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.236616 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.236770 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.236806 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:18:03.236979 9085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.233913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:34373 (local address 127.1.1.193:41737)
0620 19:18:03.234191 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:03.234199 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:03.234244 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:03.234386 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:03.234387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:03.234388 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:03.234461 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.234463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.235356 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.235357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.236120 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.236123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.236624 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.236625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.236693 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.236697 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:18:03.236747 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:18:03.236782 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":17,"thread_start_us":63,"threads_started":1}
I20260620 19:18:03.237144 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafde880bf2548b786930d780ce164a8" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6" is_pre_election: true
I20260620 19:18:03.237279 9004 raft_consensus.cc:2454] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.237351 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafde880bf2548b786930d780ce164a8" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c" is_pre_election: true
I20260620 19:18:03.237474 8864 raft_consensus.cc:2454] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.237481 8657 leader_election.cc:302] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.237607 9083 raft_consensus.cc:2781] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:03.237664 9083 raft_consensus.cc:482] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:03.237685 9083 raft_consensus.cc:3037] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.238415 9083 raft_consensus.cc:504] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.238544 9083 leader_election.cc:288] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Requested vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:03.238679 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafde880bf2548b786930d780ce164a8" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c"
I20260620 19:18:03.238763 8864 raft_consensus.cc:3037] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.238744 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aafde880bf2548b786930d780ce164a8" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6"
I20260620 19:18:03.238838 9004 raft_consensus.cc:3037] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.239431 8864 raft_consensus.cc:2454] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.239529 9004 raft_consensus.cc:2454] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.239615 8656 leader_election.cc:302] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5fd376a5a1434ab06152e51243ba0c, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.239712 9083 raft_consensus.cc:2781] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:03.239821 9083 raft_consensus.cc:686] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Becoming Leader. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Running, Role: LEADER
I20260620 19:18:03.239920 9083 consensus_queue.cc:237] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.240613 8567 catalog_manager.cc:5495] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } health_report { overall_health: UNKNOWN } } }
I20260620 19:18:03.347380 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.345386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.1.1.193:36607)
0620 19:18:03.345499 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.345501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.345516 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:18:03.345557 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.345557 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.345574 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.345645 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:03.345773 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.346199 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.346815 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.346968 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.347009 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.347149 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:03.347230 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:03.347278 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.347315 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.347328 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260620 19:18:03.348950 8864 raft_consensus.cc:1261] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.348950 9004 raft_consensus.cc:1261] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.349264 9083 consensus_queue.cc:1035] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.349416 9086 consensus_queue.cc:1035] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.353359 8567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:03.353438 8567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:03.354193 8567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Sending DeleteTablet for 3 replicas of tablet aafde880bf2548b786930d780ce164a8
I20260620 19:18:03.354545 8704 tablet_service.cc:1514] Processing DeleteTablet for tablet aafde880bf2548b786930d780ce164a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 19:18:03.354535 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet aafde880bf2548b786930d780ce164a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:45468
I20260620 19:18:03.354562 8984 tablet_service.cc:1514] Processing DeleteTablet for tablet aafde880bf2548b786930d780ce164a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39480
I20260620 19:18:03.354786 9104 tablet_replica.cc:330] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: stopping tablet replica
I20260620 19:18:03.354880 9104 raft_consensus.cc:2229] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:03.354974 9104 raft_consensus.cc:2258] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.354974 8567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:18:03.355129 8567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:03.355494 9104 ts_tablet_manager.cc:1905] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.356277 8844 tablet_service.cc:1467] Processing CreateTablet for tablet 8f19db5610684949b7365fdf075106f1 (DEFAULT_TABLE table=test-table [id=e74a99a287534497af01bfb7d330520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.356282 8984 tablet_service.cc:1467] Processing CreateTablet for tablet 8f19db5610684949b7365fdf075106f1 (DEFAULT_TABLE table=test-table [id=e74a99a287534497af01bfb7d330520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.356298 8704 tablet_service.cc:1467] Processing CreateTablet for tablet 8f19db5610684949b7365fdf075106f1 (DEFAULT_TABLE table=test-table [id=e74a99a287534497af01bfb7d330520e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.356381 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f19db5610684949b7365fdf075106f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.356382 8704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f19db5610684949b7365fdf075106f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.356382 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f19db5610684949b7365fdf075106f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.357259 9105 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Bootstrap starting.
I20260620 19:18:03.357661 9105 tablet_bootstrap.cc:654] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.357848 9106 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Bootstrap starting.
I20260620 19:18:03.358219 9106 tablet_bootstrap.cc:654] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.358560 9107 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Bootstrap starting.
I20260620 19:18:03.359042 9104 ts_tablet_manager.cc:1918] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:03.359055 9107 tablet_bootstrap.cc:654] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.359169 9104 log.cc:1196] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/aafde880bf2548b786930d780ce164a8
I20260620 19:18:03.359505 9104 ts_tablet_manager.cc:1939] T aafde880bf2548b786930d780ce164a8 P 2495de1a715346e48896bf0037eeddf6: Deleting consensus metadata
I20260620 19:18:03.359710 9106 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: No bootstrap required, opened a new log
I20260620 19:18:03.359771 9106 ts_tablet_manager.cc:1397] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:03.359874 9106 raft_consensus.cc:348] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.359957 9106 raft_consensus.cc:374] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.359982 9106 raft_consensus.cc:729] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.360056 9106 consensus_queue.cc:260] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.360203 8553 catalog_manager.cc:4841] TS 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883): tablet aafde880bf2548b786930d780ce164a8 (table test-table [id=24ab13b88c07455b9aba6832cee223eb]) successfully deleted
I20260620 19:18:03.360276 9106 ts_tablet_manager.cc:1428] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.360471 9107 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: No bootstrap required, opened a new log
I20260620 19:18:03.360590 9107 ts_tablet_manager.cc:1397] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:03.360775 9107 raft_consensus.cc:348] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.360849 9107 raft_consensus.cc:374] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.360872 9107 raft_consensus.cc:729] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5fd376a5a1434ab06152e51243ba0c, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.360916 9107 consensus_queue.cc:260] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.361038 9107 ts_tablet_manager.cc:1428] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.361613 9105 tablet_bootstrap.cc:492] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: No bootstrap required, opened a new log
I20260620 19:18:03.361660 9105 ts_tablet_manager.cc:1397] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:18:03.361974 9105 raft_consensus.cc:348] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.362052 9105 raft_consensus.cc:374] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.362067 9105 raft_consensus.cc:729] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.362116 9105 consensus_queue.cc:260] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.362211 9105 ts_tablet_manager.cc:1428] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:18:03.365164 9103 tablet_replica.cc:330] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: stopping tablet replica
I20260620 19:18:03.365178 9102 tablet_replica.cc:330] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: stopping tablet replica
I20260620 19:18:03.365273 9103 raft_consensus.cc:2229] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:18:03.365273 9102 raft_consensus.cc:2229] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:03.365361 9103 raft_consensus.cc:2258] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.365373 9102 raft_consensus.cc:2258] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.365701 9102 ts_tablet_manager.cc:1905] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.366982 9102 ts_tablet_manager.cc:1918] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:03.367045 9102 log.cc:1196] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/aafde880bf2548b786930d780ce164a8
I20260620 19:18:03.367249 9102 ts_tablet_manager.cc:1939] T aafde880bf2548b786930d780ce164a8 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting consensus metadata
I20260620 19:18:03.367602 8552 catalog_manager.cc:4841] TS 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373): tablet aafde880bf2548b786930d780ce164a8 (table test-table [id=24ab13b88c07455b9aba6832cee223eb]) successfully deleted
I20260620 19:18:03.370913 9103 ts_tablet_manager.cc:1905] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.371970 9103 ts_tablet_manager.cc:1918] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:03.372066 9103 log.cc:1196] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/aafde880bf2548b786930d780ce164a8
I20260620 19:18:03.372345 9103 ts_tablet_manager.cc:1939] T aafde880bf2548b786930d780ce164a8 P f4d86a23204d4b13a757d9a0de4bed00: Deleting consensus metadata
I20260620 19:18:03.372717 8553 catalog_manager.cc:4841] TS f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607): tablet aafde880bf2548b786930d780ce164a8 (table test-table [id=24ab13b88c07455b9aba6832cee223eb]) successfully deleted
I20260620 19:18:03.375489 9086 raft_consensus.cc:482] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:03.375576 9086 raft_consensus.cc:504] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.375702 9086 leader_election.cc:288] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:03.375890 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f19db5610684949b7365fdf075106f1" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c" is_pre_election: true
I20260620 19:18:03.375978 8864 raft_consensus.cc:2454] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.375994 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f19db5610684949b7365fdf075106f1" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6" is_pre_election: true
I20260620 19:18:03.376060 9004 raft_consensus.cc:2454] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.376106 8656 leader_election.cc:302] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5fd376a5a1434ab06152e51243ba0c, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.376174 9086 raft_consensus.cc:2781] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:03.376221 9086 raft_consensus.cc:482] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:03.376240 9086 raft_consensus.cc:3037] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.376844 9086 raft_consensus.cc:504] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.376960 9086 leader_election.cc:288] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Requested vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:03.377105 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f19db5610684949b7365fdf075106f1" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6"
I20260620 19:18:03.377147 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f19db5610684949b7365fdf075106f1" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c"
I20260620 19:18:03.377183 9004 raft_consensus.cc:3037] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.377220 8864 raft_consensus.cc:3037] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.377663 9004 raft_consensus.cc:2454] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.377805 8657 leader_election.cc:302] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.377890 9086 raft_consensus.cc:2781] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:03.377928 9086 raft_consensus.cc:686] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Becoming Leader. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Running, Role: LEADER
I20260620 19:18:03.377956 9086 consensus_queue.cc:237] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.378064 8864 raft_consensus.cc:2454] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.378402 8567 catalog_manager.cc:5495] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:18:03 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983082, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.254@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:03.416760 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.409616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35619 (local address 127.0.0.1:53718)
0620 19:18:03.409770 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:03.409797 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:03.409861 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:03.409991 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:03.409992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:03.410353 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:03.410404 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.410407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.411021 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.411022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.411830 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.411832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.412164 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.412165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.412222 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.412705 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:03.412710 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:03.414096 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:03.415800 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:03.415805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:03.415807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:03.416035 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:03.416220 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:03.416220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:03.416220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:03.416270 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:03.416526 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:03.416531 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:03.416566 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:18:03.416616 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:18:03.416695 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260620 19:18:03.416863 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.409702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.1.1.254:35619)
0620 19:18:03.409860 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.409862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.409870 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:03.409906 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.409906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.409915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.409962 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:03.410482 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.410941 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.411910 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.412090 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.412136 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.414170 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:03.414172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:03.414172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:03.414178 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:03.415720 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:03.416103 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:03.416104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:03.416104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:03.416159 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:03.416342 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:03.416342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:03.416343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:03.416466 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:03.416530 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.416655 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.416760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":128,"thread_start_us":107,"threads_started":1}
I20260620 19:18:03.418669 8567 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:53718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:18:03.418876 8567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:03.420286 8704 tablet_service.cc:1467] Processing CreateTablet for tablet 671da8e6da944ec58973073d97748b6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=25c75178be25464a8f345f3a86a2c37d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:18:03.420328 8844 tablet_service.cc:1467] Processing CreateTablet for tablet 671da8e6da944ec58973073d97748b6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=25c75178be25464a8f345f3a86a2c37d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:18:03.420425 8704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671da8e6da944ec58973073d97748b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.420456 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671da8e6da944ec58973073d97748b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.421474 9106 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: Bootstrap starting.
I20260620 19:18:03.421581 8984 tablet_service.cc:1467] Processing CreateTablet for tablet 671da8e6da944ec58973073d97748b6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=25c75178be25464a8f345f3a86a2c37d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:18:03.421703 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 671da8e6da944ec58973073d97748b6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.421908 9106 tablet_bootstrap.cc:654] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.422593 9106 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: No bootstrap required, opened a new log
I20260620 19:18:03.422643 9106 ts_tablet_manager.cc:1397] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:03.422797 9106 raft_consensus.cc:348] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.422878 9106 raft_consensus.cc:374] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.422904 9106 raft_consensus.cc:729] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.422942 9106 consensus_queue.cc:260] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.423039 9106 ts_tablet_manager.cc:1428] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:03.423144 9083 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.423202 9107 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: Bootstrap starting.
I20260620 19:18:03.423210 9083 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.423596 9107 tablet_bootstrap.cc:654] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.423935 9105 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: Bootstrap starting.
I20260620 19:18:03.424314 9105 tablet_bootstrap.cc:654] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.425128 9105 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: No bootstrap required, opened a new log
I20260620 19:18:03.425181 9105 ts_tablet_manager.cc:1397] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:03.425312 9105 raft_consensus.cc:348] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.425364 9105 raft_consensus.cc:374] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.425376 9105 raft_consensus.cc:729] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.425432 9105 consensus_queue.cc:260] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.425542 9105 ts_tablet_manager.cc:1428] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.425602 9092 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.425680 9092 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.425743 9107 tablet_bootstrap.cc:492] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: No bootstrap required, opened a new log
I20260620 19:18:03.425781 9107 ts_tablet_manager.cc:1397] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:18:03.425897 9107 raft_consensus.cc:348] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.425995 9107 raft_consensus.cc:374] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.426017 9107 raft_consensus.cc:729] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5fd376a5a1434ab06152e51243ba0c, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.426059 9107 consensus_queue.cc:260] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.426160 9107 ts_tablet_manager.cc:1428] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.426209 9093 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.426280 9093 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.750170 9083 raft_consensus.cc:482] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:03.750247 9083 raft_consensus.cc:504] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.750420 9083 leader_election.cc:288] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883), 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373)
I20260620 19:18:03.750633 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671da8e6da944ec58973073d97748b6e" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6" is_pre_election: true
I20260620 19:18:03.750638 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671da8e6da944ec58973073d97748b6e" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c" is_pre_election: true
I20260620 19:18:03.750723 9004 raft_consensus.cc:2454] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.750723 8864 raft_consensus.cc:2454] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:03.750842 8657 leader_election.cc:302] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.750958 9083 raft_consensus.cc:2781] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:03.750993 9083 raft_consensus.cc:482] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:03.751005 9083 raft_consensus.cc:3037] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.751647 9083 raft_consensus.cc:504] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.751775 9083 leader_election.cc:288] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Requested vote from peers 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883), 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373)
I20260620 19:18:03.751915 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671da8e6da944ec58973073d97748b6e" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c"
I20260620 19:18:03.751919 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "671da8e6da944ec58973073d97748b6e" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6"
I20260620 19:18:03.751984 8864 raft_consensus.cc:3037] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.751986 9004 raft_consensus.cc:3037] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:03.752658 9004 raft_consensus.cc:2454] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.752658 8864 raft_consensus.cc:2454] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:03.752815 8657 leader_election.cc:302] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:03.752923 9083 raft_consensus.cc:2781] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:03.753002 9083 raft_consensus.cc:686] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Becoming Leader. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Running, Role: LEADER
I20260620 19:18:03.753108 9083 consensus_queue.cc:237] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } }
I20260620 19:18:03.753397 9086 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: TxnStatusTablet state changed. Reason: New leader f4d86a23204d4b13a757d9a0de4bed00. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.753480 9086 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:18:03.753667 9123 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:18:03.753783 9123 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:18:03.753707 8567 catalog_manager.cc:5495] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } health_report { overall_health: UNKNOWN } } }
I20260620 19:18:03.781349 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.778956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.1.1.193:36607)
0620 19:18:03.779035 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.779036 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.779043 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:18:03.779112 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.779112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.779121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.779164 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:03.779298 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.779733 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.780747 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.780904 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.780949 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.781173 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:03.781195 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:03.781240 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.781292 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.781306 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 19:18:03.781399 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.778900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.0.0.1:50068)
0620 19:18:03.778994 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:03.779004 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:03.779044 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:03.779172 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:03.779173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:03.779174 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:18:03.779223 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.779226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.779803 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.779804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.780672 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.780675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.780979 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.780980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.781031 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.781046 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:18:03.781241 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:18:03.781245 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:18:03.781290 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:18:03.781330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:18:03.782332 9004 raft_consensus.cc:1261] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.782385 8864 raft_consensus.cc:1261] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.782507 9086 consensus_queue.cc:1035] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.782665 9086 consensus_queue.cc:1035] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.782894 9092 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: TxnStatusTablet state changed. Reason: New leader f4d86a23204d4b13a757d9a0de4bed00. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.782971 9092 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.783417 9086 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.783497 9086 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:18:03.783614 9091 mvcc.cc:205] Tried to move back new op lower bound from 7299002711171117056 to 7299002711053139968. Current Snapshot: MvccSnapshot[applied={T|T < 7299002711171117056 or (T in {7299002711171117056})}]
I20260620 19:18:03.783733 9092 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.783771 9094 mvcc.cc:205] Tried to move back new op lower bound from 7299002711171117056 to 7299002711053139968. Current Snapshot: MvccSnapshot[applied={T|T < 7299002711171117056}]
I20260620 19:18:03.783799 9092 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 2495de1a715346e48896bf0037eeddf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.784446 9093 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: TxnStatusTablet state changed. Reason: New leader f4d86a23204d4b13a757d9a0de4bed00. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.784545 9093 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.785542 9083 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.785622 9083 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P f4d86a23204d4b13a757d9a0de4bed00: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:18:03.785768 9095 mvcc.cc:205] Tried to move back new op lower bound from 7299002711171117056 to 7299002711053139968. Current Snapshot: MvccSnapshot[applied={T|T < 7299002711171117056}]
I20260620 19:18:03.785876 9093 tablet_replica.cc:439] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } }
I20260620 19:18:03.785969 9093 tablet_replica.cc:442] T 671da8e6da944ec58973073d97748b6e P 5a5fd376a5a1434ab06152e51243ba0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:18:03.788856 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.786650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.1.1.193:52279)
0620 19:18:03.786763 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:03.786774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:03.786838 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:03.786953 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:03.786953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:03.786954 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:18:03.786984 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.786985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.787704 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.787705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.788156 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:03.788157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.788544 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.788545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:03.788576 (+ 31us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.788586 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:18:03.788722 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:18:03.788724 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:03.788749 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 19:18:03.788776 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":64,"threads_started":1}
I20260620 19:18:03.788897 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.786675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52279 (local address 127.1.1.193:36607)
0620 19:18:03.786701 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.786703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.786819 (+ 116us) server_negotiation.cc:407] Connection header received
0620 19:18:03.786889 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.786890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.786898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.786953 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:03.787019 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.787627 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.788228 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.788481 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.788547 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.788652 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:03.788676 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:03.788717 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.788762 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.788780 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12}
I20260620 19:18:03.790885 8864 raft_consensus.cc:1261] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.791009 9004 raft_consensus.cc:1261] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:03.791074 9086 consensus_queue.cc:1035] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.791155 9131 consensus_queue.cc:1035] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:03.803169 8565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:03.803247 8565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:03.803718 8565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Sending DeleteTablet for 3 replicas of tablet 8f19db5610684949b7365fdf075106f1
I20260620 19:18:03.803922 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f19db5610684949b7365fdf075106f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 19:18:03.803959 8984 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f19db5610684949b7365fdf075106f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39480
I20260620 19:18:03.803997 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f19db5610684949b7365fdf075106f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:03 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:45468
I20260620 19:18:03.804029 9104 tablet_replica.cc:330] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: stopping tablet replica
I20260620 19:18:03.804028 9103 tablet_replica.cc:330] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: stopping tablet replica
I20260620 19:18:03.804066 9104 raft_consensus.cc:2229] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:03.804075 9103 raft_consensus.cc:2229] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:18:03.804098 9104 raft_consensus.cc:2258] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.804116 9103 raft_consensus.cc:2258] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.804157 9102 tablet_replica.cc:330] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: stopping tablet replica
I20260620 19:18:03.804205 9102 raft_consensus.cc:2229] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:03.804236 9102 raft_consensus.cc:2258] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:03.804517 9102 ts_tablet_manager.cc:1905] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.804561 9104 ts_tablet_manager.cc:1905] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.804557 9103 ts_tablet_manager.cc:1905] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:03.806133 9102 ts_tablet_manager.cc:1918] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:18:03.806144 9104 ts_tablet_manager.cc:1918] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:18:03.806181 9102 log.cc:1196] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/8f19db5610684949b7365fdf075106f1
I20260620 19:18:03.806188 9104 log.cc:1196] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/8f19db5610684949b7365fdf075106f1
I20260620 19:18:03.806389 9102 ts_tablet_manager.cc:1939] T 8f19db5610684949b7365fdf075106f1 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting consensus metadata
I20260620 19:18:03.806407 9104 ts_tablet_manager.cc:1939] T 8f19db5610684949b7365fdf075106f1 P 2495de1a715346e48896bf0037eeddf6: Deleting consensus metadata
I20260620 19:18:03.806797 8553 catalog_manager.cc:4841] TS 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883): tablet 8f19db5610684949b7365fdf075106f1 (table test-table [id=e74a99a287534497af01bfb7d330520e]) successfully deleted
I20260620 19:18:03.808878 8552 catalog_manager.cc:4841] TS 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373): tablet 8f19db5610684949b7365fdf075106f1 (table test-table [id=e74a99a287534497af01bfb7d330520e]) successfully deleted
I20260620 19:18:03.808952 9103 ts_tablet_manager.cc:1918] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:18:03.809003 9103 log.cc:1196] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/8f19db5610684949b7365fdf075106f1
I20260620 19:18:03.809216 9103 ts_tablet_manager.cc:1939] T 8f19db5610684949b7365fdf075106f1 P f4d86a23204d4b13a757d9a0de4bed00: Deleting consensus metadata
I20260620 19:18:03.809494 8553 catalog_manager.cc:4841] TS f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607): tablet 8f19db5610684949b7365fdf075106f1 (table test-table [id=e74a99a287534497af01bfb7d330520e]) successfully deleted
Jun 20 19:18:03 dist-test-slave-mwkb krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:18:03 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:03 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:18:03.827114 9118 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:03.821473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.1.1.254:35619)
0620 19:18:03.821521 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.821522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.821632 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:18:03.821689 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.821689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.821698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.821739 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:03.822341 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.822796 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.823673 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.823828 (+ 155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.823869 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.827023 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53730: BlockingRecv error: recv got EOF from 127.0.0.1:53730 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 19:18:03 dist-test-slave-mwkb krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM
I20260620 19:18:03.834437 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:03.827891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.1.1.254:35619)
0620 19:18:03.827935 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:03.827937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:03.827968 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:18:03.828074 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:03.828076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:03.828085 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:03.828132 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:03.828572 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.829017 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.829754 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:03.829949 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:03.830037 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:03.832030 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:03.832032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:03.832032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:03.832039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:03.833434 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:03.833742 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:03.833743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:03.833744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:03.833798 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:03.833991 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:03.833992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:03.833992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:03.834088 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:03.834184 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:03.834278 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:18:03.834348 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:18:03.835861 8565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:18:03.836037 8565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:03.837281 8703 tablet_service.cc:1467] Processing CreateTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 (DEFAULT_TABLE table=test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.837277 8984 tablet_service.cc:1467] Processing CreateTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 (DEFAULT_TABLE table=test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.837416 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94859b8486264f8f8dd5deb1d28faeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.837417 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94859b8486264f8f8dd5deb1d28faeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.838414 9106 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Bootstrap starting.
I20260620 19:18:03.838467 8844 tablet_service.cc:1467] Processing CreateTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 (DEFAULT_TABLE table=test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:03.838575 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94859b8486264f8f8dd5deb1d28faeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:03.838634 9105 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Bootstrap starting.
I20260620 19:18:03.838696 9106 tablet_bootstrap.cc:654] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.838989 9105 tablet_bootstrap.cc:654] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.839725 9105 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: No bootstrap required, opened a new log
I20260620 19:18:03.839802 9105 ts_tablet_manager.cc:1397] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:03.839824 9107 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Bootstrap starting.
I20260620 19:18:03.839938 9105 raft_consensus.cc:348] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.839999 9105 raft_consensus.cc:374] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.840019 9105 raft_consensus.cc:729] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.840062 9105 consensus_queue.cc:260] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.840148 9105 ts_tablet_manager.cc:1428] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.840173 9107 tablet_bootstrap.cc:654] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:03.840642 9106 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: No bootstrap required, opened a new log
I20260620 19:18:03.840689 9106 ts_tablet_manager.cc:1397] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:03.840790 9106 raft_consensus.cc:348] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.840848 9106 raft_consensus.cc:374] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.840879 9106 raft_consensus.cc:729] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.840915 9106 consensus_queue.cc:260] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.841001 9106 ts_tablet_manager.cc:1428] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:03.841365 9107 tablet_bootstrap.cc:492] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: No bootstrap required, opened a new log
I20260620 19:18:03.841415 9107 ts_tablet_manager.cc:1397] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:03.841562 9107 raft_consensus.cc:348] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.841636 9107 raft_consensus.cc:374] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:03.841660 9107 raft_consensus.cc:729] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5fd376a5a1434ab06152e51243ba0c, State: Initialized, Role: FOLLOWER
I20260620 19:18:03.841701 9107 consensus_queue.cc:260] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:03.841785 9107 ts_tablet_manager.cc:1428] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:04.072067 9134 raft_consensus.cc:482] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:04.072156 9134 raft_consensus.cc:504] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.072351 9134 leader_election.cc:288] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:04.072546 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94859b8486264f8f8dd5deb1d28faeb0" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6" is_pre_election: true
I20260620 19:18:04.072556 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94859b8486264f8f8dd5deb1d28faeb0" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c" is_pre_election: true
I20260620 19:18:04.072633 9004 raft_consensus.cc:2454] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:04.072638 8864 raft_consensus.cc:2454] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 0.
I20260620 19:18:04.072854 8656 leader_election.cc:302] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5fd376a5a1434ab06152e51243ba0c, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:04.072966 9134 raft_consensus.cc:2781] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:04.073010 9134 raft_consensus.cc:482] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:04.073027 9134 raft_consensus.cc:3037] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.073694 9134 raft_consensus.cc:504] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.073817 9134 leader_election.cc:288] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Requested vote from peers 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373), 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883)
I20260620 19:18:04.073942 9004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94859b8486264f8f8dd5deb1d28faeb0" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495de1a715346e48896bf0037eeddf6"
I20260620 19:18:04.073964 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94859b8486264f8f8dd5deb1d28faeb0" candidate_uuid: "f4d86a23204d4b13a757d9a0de4bed00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c"
I20260620 19:18:04.074016 8864 raft_consensus.cc:3037] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.074028 9004 raft_consensus.cc:3037] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.074540 8864 raft_consensus.cc:2454] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:04.074615 9004 raft_consensus.cc:2454] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4d86a23204d4b13a757d9a0de4bed00 in term 1.
I20260620 19:18:04.074683 8656 leader_election.cc:302] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a5fd376a5a1434ab06152e51243ba0c, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:04.074833 9134 raft_consensus.cc:2781] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:04.074885 9134 raft_consensus.cc:686] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Becoming Leader. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Running, Role: LEADER
I20260620 19:18:04.074930 9134 consensus_queue.cc:237] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.075502 8565 catalog_manager.cc:5495] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "f4d86a23204d4b13a757d9a0de4bed00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } health_report { overall_health: UNKNOWN } } }
I20260620 19:18:04.124243 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.121670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.1.1.193:36607)
0620 19:18:04.121712 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.121713 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.121722 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:04.121776 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.121777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.121787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.121832 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:04.121988 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.122521 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.123503 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.123718 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.123815 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.123983 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.124022 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.124088 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.124144 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.124163 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:18:04.124928 8864 raft_consensus.cc:1261] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:04.124953 9004 raft_consensus.cc:1261] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Refusing update from remote peer f4d86a23204d4b13a757d9a0de4bed00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:04.125190 9131 consensus_queue.cc:1035] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:04.125288 9134 consensus_queue.cc:1035] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:04.127736 8565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:04.127823 8565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:04.128396 8565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Sending DeleteTablet for 3 replicas of tablet 94859b8486264f8f8dd5deb1d28faeb0
I20260620 19:18:04.128660 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 19:18:04.128700 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:45468
I20260620 19:18:04.128705 8984 tablet_service.cc:1514] Processing DeleteTablet for tablet 94859b8486264f8f8dd5deb1d28faeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39480
I20260620 19:18:04.128750 9103 tablet_replica.cc:330] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: stopping tablet replica
I20260620 19:18:04.128772 9102 tablet_replica.cc:330] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: stopping tablet replica
I20260620 19:18:04.128803 9103 raft_consensus.cc:2229] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:18:04.128811 9102 raft_consensus.cc:2229] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:04.128849 9102 raft_consensus.cc:2258] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.128856 9103 raft_consensus.cc:2258] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.128926 9104 tablet_replica.cc:330] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: stopping tablet replica
I20260620 19:18:04.128983 9104 raft_consensus.cc:2229] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:04.129031 9104 raft_consensus.cc:2258] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.129231 8565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:18:04.129302 9102 ts_tablet_manager.cc:1905] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:18:04.129341 8565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:04.129709 9104 ts_tablet_manager.cc:1905] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:04.130393 8703 tablet_service.cc:1467] Processing CreateTablet for tablet c71336c52cf044e7991ef4a34bded041 (DEFAULT_TABLE table=test-table [id=3635aaffad864311975652a120cd6d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:04.130462 8844 tablet_service.cc:1467] Processing CreateTablet for tablet c71336c52cf044e7991ef4a34bded041 (DEFAULT_TABLE table=test-table [id=3635aaffad864311975652a120cd6d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:04.130502 8703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71336c52cf044e7991ef4a34bded041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:04.130550 8844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71336c52cf044e7991ef4a34bded041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:04.130856 9102 ts_tablet_manager.cc:1918] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:04.130904 9102 log.cc:1196] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/94859b8486264f8f8dd5deb1d28faeb0
I20260620 19:18:04.131116 9102 ts_tablet_manager.cc:1939] T 94859b8486264f8f8dd5deb1d28faeb0 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting consensus metadata
I20260620 19:18:04.131213 9104 ts_tablet_manager.cc:1918] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:04.131268 9104 log.cc:1196] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/94859b8486264f8f8dd5deb1d28faeb0
I20260620 19:18:04.131440 8984 tablet_service.cc:1467] Processing CreateTablet for tablet c71336c52cf044e7991ef4a34bded041 (DEFAULT_TABLE table=test-table [id=3635aaffad864311975652a120cd6d07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:04.131551 8984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c71336c52cf044e7991ef4a34bded041. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:04.131669 9107 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Bootstrap starting.
I20260620 19:18:04.131731 8552 catalog_manager.cc:4841] TS 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373): tablet 94859b8486264f8f8dd5deb1d28faeb0 (table test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]) successfully deleted
I20260620 19:18:04.132047 9107 tablet_bootstrap.cc:654] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:04.132558 9106 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: Bootstrap starting.
I20260620 19:18:04.132709 9105 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Bootstrap starting.
I20260620 19:18:04.131440 9104 ts_tablet_manager.cc:1939] T 94859b8486264f8f8dd5deb1d28faeb0 P 2495de1a715346e48896bf0037eeddf6: Deleting consensus metadata
I20260620 19:18:04.132843 9106 tablet_bootstrap.cc:654] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:04.133117 9105 tablet_bootstrap.cc:654] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:04.133122 8553 catalog_manager.cc:4841] TS 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883): tablet 94859b8486264f8f8dd5deb1d28faeb0 (table test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]) successfully deleted
I20260620 19:18:04.133179 9107 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: No bootstrap required, opened a new log
I20260620 19:18:04.133224 9107 ts_tablet_manager.cc:1397] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:04.133342 9103 ts_tablet_manager.cc:1905] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:04.133352 9107 raft_consensus.cc:348] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.133391 9107 raft_consensus.cc:374] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:04.133402 9107 raft_consensus.cc:729] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5fd376a5a1434ab06152e51243ba0c, State: Initialized, Role: FOLLOWER
I20260620 19:18:04.133453 9107 consensus_queue.cc:260] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.133520 9107 ts_tablet_manager.cc:1428] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:04.133847 9106 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: No bootstrap required, opened a new log
I20260620 19:18:04.133888 9106 ts_tablet_manager.cc:1397] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:18:04.133976 9106 raft_consensus.cc:348] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.134037 9106 raft_consensus.cc:374] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:04.134057 9106 raft_consensus.cc:729] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4d86a23204d4b13a757d9a0de4bed00, State: Initialized, Role: FOLLOWER
I20260620 19:18:04.134094 9106 consensus_queue.cc:260] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.134192 9106 ts_tablet_manager.cc:1428] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:04.134603 9103 ts_tablet_manager.cc:1918] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:04.134658 9103 log.cc:1196] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/94859b8486264f8f8dd5deb1d28faeb0
I20260620 19:18:04.134886 9103 ts_tablet_manager.cc:1939] T 94859b8486264f8f8dd5deb1d28faeb0 P f4d86a23204d4b13a757d9a0de4bed00: Deleting consensus metadata
I20260620 19:18:04.135196 8553 catalog_manager.cc:4841] TS f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607): tablet 94859b8486264f8f8dd5deb1d28faeb0 (table test-table [id=82fb0e01312e4c99a8a9a41f28d7479a]) successfully deleted
I20260620 19:18:04.135439 9105 tablet_bootstrap.cc:492] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: No bootstrap required, opened a new log
I20260620 19:18:04.135479 9105 ts_tablet_manager.cc:1397] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:18:04.135629 9105 raft_consensus.cc:348] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.135694 9105 raft_consensus.cc:374] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:04.135708 9105 raft_consensus.cc:729] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Initialized, Role: FOLLOWER
I20260620 19:18:04.135774 9105 consensus_queue.cc:260] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.135881 9105 ts_tablet_manager.cc:1428] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:04.138015 9092 raft_consensus.cc:482] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:04.138068 9092 raft_consensus.cc:504] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.138374 9092 leader_election.cc:288] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607), 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373)
I20260620 19:18:04.141320 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.138653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40507 (local address 127.1.1.193:36607)
0620 19:18:04.138696 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.138698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.138759 (+ 61us) server_negotiation.cc:407] Connection header received
0620 19:18:04.138808 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.138808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.138818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.138883 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:04.139083 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.139531 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.140479 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.140969 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.141048 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.141058 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.141235 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.141268 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:18:04.141336 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.138527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.1.1.195:40507)
0620 19:18:04.138700 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:04.138710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:04.138770 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:04.138887 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:04.138888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:04.138904 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:04.138964 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.138966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.139638 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.139639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.140396 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.140397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.141088 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.141089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.141151 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.141154 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:04.141228 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:18:04.141264 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":51,"threads_started":1}
I20260620 19:18:04.141507 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.138607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:60277 (local address 127.1.1.194:34373)
0620 19:18:04.138707 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.138710 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.138879 (+ 169us) server_negotiation.cc:407] Connection header received
0620 19:18:04.138962 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.138963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.139007 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.139072 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:04.139214 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.139743 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.140637 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.141171 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.141249 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.141274 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.141382 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.141422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
I20260620 19:18:04.141682 8724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71336c52cf044e7991ef4a34bded041" candidate_uuid: "2495de1a715346e48896bf0037eeddf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d86a23204d4b13a757d9a0de4bed00" is_pre_election: true
I20260620 19:18:04.141772 8724 raft_consensus.cc:2454] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2495de1a715346e48896bf0037eeddf6 in term 0.
I20260620 19:18:04.141930 8937 leader_election.cc:302] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:04.142019 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.138527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:34373 (local address 127.1.1.195:60277)
0620 19:18:04.138824 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:04.138831 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:04.138896 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:04.139075 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:04.139076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:04.139077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:04.139137 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.139139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.139839 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.139840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.140539 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.140541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.141252 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.141253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.141317 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.141320 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:04.141850 (+ 530us) client_negotiation.cc:240] Negotiation successful
0620 19:18:04.141908 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":15,"thread_start_us":39,"threads_started":1}
I20260620 19:18:04.142091 9092 raft_consensus.cc:2781] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:04.142127 9092 raft_consensus.cc:482] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:04.142154 9092 raft_consensus.cc:3037] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.142617 9092 raft_consensus.cc:504] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.142730 9092 leader_election.cc:288] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [CANDIDATE]: Term 1 election: Requested vote from peers f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607), 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373)
I20260620 19:18:04.142865 8724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71336c52cf044e7991ef4a34bded041" candidate_uuid: "2495de1a715346e48896bf0037eeddf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4d86a23204d4b13a757d9a0de4bed00"
I20260620 19:18:04.142920 8724 raft_consensus.cc:3037] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.142913 8864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71336c52cf044e7991ef4a34bded041" candidate_uuid: "2495de1a715346e48896bf0037eeddf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c" is_pre_election: true
I20260620 19:18:04.142967 8864 raft_consensus.cc:2454] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2495de1a715346e48896bf0037eeddf6 in term 0.
I20260620 19:18:04.142982 8863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c71336c52cf044e7991ef4a34bded041" candidate_uuid: "2495de1a715346e48896bf0037eeddf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a5fd376a5a1434ab06152e51243ba0c"
I20260620 19:18:04.143066 8863 raft_consensus.cc:3037] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:04.143453 8724 raft_consensus.cc:2454] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2495de1a715346e48896bf0037eeddf6 in term 1.
I20260620 19:18:04.143623 8863 raft_consensus.cc:2454] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2495de1a715346e48896bf0037eeddf6 in term 1.
I20260620 19:18:04.143642 8937 leader_election.cc:302] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495de1a715346e48896bf0037eeddf6, f4d86a23204d4b13a757d9a0de4bed00; no voters:
I20260620 19:18:04.143715 9092 raft_consensus.cc:2781] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:04.143848 9092 raft_consensus.cc:686] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 1 LEADER]: Becoming Leader. State: Replica: 2495de1a715346e48896bf0037eeddf6, State: Running, Role: LEADER
I20260620 19:18:04.143923 9092 consensus_queue.cc:237] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } }
I20260620 19:18:04.144429 8566 catalog_manager.cc:5495] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2495de1a715346e48896bf0037eeddf6 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "2495de1a715346e48896bf0037eeddf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495de1a715346e48896bf0037eeddf6" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32883 } health_report { overall_health: HEALTHY } } }
I20260620 19:18:04.152968 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.150851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.1.1.195:32883)
0620 19:18:04.150995 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.150996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.151001 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:18:04.151023 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.151024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.151037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.151112 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:04.151194 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.151615 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.152238 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.152429 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.152507 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.152666 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.152762 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.152832 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.152872 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.152888 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 19:18:04.157230 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.155031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:42351 (local address 127.1.1.193:36607)
0620 19:18:04.155094 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.155096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.155181 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:18:04.155241 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.155242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.155251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.155304 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:04.155435 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.155867 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.156595 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.156774 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.156834 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.157007 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.157042 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.157117 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.157164 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.157179 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:18:04.157262 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.154956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.1.1.195:42351)
0620 19:18:04.155111 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:04.155120 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:04.155195 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:04.155308 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:04.155308 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:04.155310 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:18:04.155365 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.155366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.155946 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.155947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.156492 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.156494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.156860 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.156860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.156920 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.156929 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.157120 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.157122 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:04.157162 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 19:18:04.157195 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 19:18:04.158785 8863 raft_consensus.cc:1261] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Refusing update from remote peer 2495de1a715346e48896bf0037eeddf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:04.158910 8724 raft_consensus.cc:1261] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Refusing update from remote peer 2495de1a715346e48896bf0037eeddf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:04.159040 9092 consensus_queue.cc:1035] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a5fd376a5a1434ab06152e51243ba0c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 34373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:04.159183 9161 consensus_queue.cc:1035] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4d86a23204d4b13a757d9a0de4bed00" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:04.164364 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.162741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:50023 (local address 127.1.1.195:32883)
0620 19:18:04.162818 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:04.162819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:04.162825 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:04.162844 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:04.162844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:04.162851 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:04.162892 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:04.162972 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.163446 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.163989 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.164129 (+ 140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.164170 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.164251 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.164267 (+ 16us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.164294 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:04.164320 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:18:04.164330 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 19:18:04.164381 9130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.162707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:32883 (local address 127.1.1.193:50023)
0620 19:18:04.162744 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:04.162751 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:04.162780 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:04.162894 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:04.162895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:04.162896 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:18:04.162920 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.162921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.163498 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.163499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.163927 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:04.163928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:04.164180 (+ 252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:04.164181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:04.164211 (+ 30us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:04.164215 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:18:04.164294 (+ 79us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:18:04.164295 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:18:04.164320 (+ 25us) client_negotiation.cc:240] Negotiation successful
0620 19:18:04.164344 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:18:04.171051 8566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:04.171123 8566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:04.171578 8566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6152c0764c664a7a9a25939f88464d70: Sending DeleteTablet for 3 replicas of tablet c71336c52cf044e7991ef4a34bded041
I20260620 19:18:04.171816 8844 tablet_service.cc:1514] Processing DeleteTablet for tablet c71336c52cf044e7991ef4a34bded041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:45468
I20260620 19:18:04.171885 8703 tablet_service.cc:1514] Processing DeleteTablet for tablet c71336c52cf044e7991ef4a34bded041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50050
I20260620 19:18:04.171948 9103 tablet_replica.cc:330] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00: stopping tablet replica
I20260620 19:18:04.171960 9102 tablet_replica.cc:330] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: stopping tablet replica
I20260620 19:18:04.171990 9103 raft_consensus.cc:2229] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:04.172014 9102 raft_consensus.cc:2229] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:04.172024 9103 raft_consensus.cc:2258] T c71336c52cf044e7991ef4a34bded041 P f4d86a23204d4b13a757d9a0de4bed00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.172048 9102 raft_consensus.cc:2258] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.172209 8984 tablet_service.cc:1514] Processing DeleteTablet for tablet c71336c52cf044e7991ef4a34bded041 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:04 UTC) from {username='oryx', principal='oryx/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39480
I20260620 19:18:04.172219 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8639
I20260620 19:18:04.172273 9104 tablet_replica.cc:330] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: stopping tablet replica
I20260620 19:18:04.172307 9104 raft_consensus.cc:2229] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:18:04.172326 9102 ts_tablet_manager.cc:1905] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:04.172363 9104 raft_consensus.cc:2258] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:18:04.172919 9104 ts_tablet_manager.cc:1905] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:04.173549 9102 ts_tablet_manager.cc:1918] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:18:04.173594 9102 log.cc:1196] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/c71336c52cf044e7991ef4a34bded041
I20260620 19:18:04.173767 9102 ts_tablet_manager.cc:1939] T c71336c52cf044e7991ef4a34bded041 P 5a5fd376a5a1434ab06152e51243ba0c: Deleting consensus metadata
I20260620 19:18:04.174140 9104 ts_tablet_manager.cc:1918] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:18:04.174232 9104 log.cc:1196] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/c71336c52cf044e7991ef4a34bded041
I20260620 19:18:04.174407 9104 ts_tablet_manager.cc:1939] T c71336c52cf044e7991ef4a34bded041 P 2495de1a715346e48896bf0037eeddf6: Deleting consensus metadata
I20260620 19:18:04.174762 8553 catalog_manager.cc:4841] TS 2495de1a715346e48896bf0037eeddf6 (127.1.1.195:32883): tablet c71336c52cf044e7991ef4a34bded041 (table test-table [id=3635aaffad864311975652a120cd6d07]) successfully deleted
I20260620 19:18:04.181172 8552 catalog_manager.cc:4841] TS 5a5fd376a5a1434ab06152e51243ba0c (127.1.1.194:34373): tablet c71336c52cf044e7991ef4a34bded041 (table test-table [id=3635aaffad864311975652a120cd6d07]) successfully deleted
W20260620 19:18:04.181528 8553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:36607 (error 108)
I20260620 19:18:04.181665 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8779
I20260620 19:18:04.182802 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:04.182596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:36607 (local address 127.0.0.1:50082)
0620 19:18:04.182721 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:04.182769 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:36607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
W20260620 19:18:04.182951 8553 catalog_manager.cc:4584] TS f4d86a23204d4b13a757d9a0de4bed00 (127.1.1.193:36607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c71336c52cf044e7991ef4a34bded041: Network error: Client connection negotiation failed: client connection to 127.1.1.193:36607: connect: Connection refused (error 111)
I20260620 19:18:04.186172 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8919
I20260620 19:18:04.190769 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 8535
2026-06-20T19:18:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3073 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:18:04 dist-test-slave-mwkb krb5kdc[9175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:18:04 dist-test-slave-mwkb krb5kdc[9175](info): set up 2 sockets
Jun 20 19:18:04 dist-test-slave-mwkb krb5kdc[9175](info): commencing operation
krb5kdc: starting...
W20260620 19:18:05.165226 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:18:05 dist-test-slave-mwkb krb5kdc[9175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:18:05.265574 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46707
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:05.336277 9188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:05.336474 9188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:05.336506 9188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:05.337606 9188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:05.337666 9188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:05.337689 9188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:05.337713 9188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:05.337742 9188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:05.339111 9188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:05.339370 9188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:05.339589 9188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:05.344205 9194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:05.344213 9196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:05.344214 9193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:05.344272 9188 server_base.cc:1034] running on GCE node
I20260620 19:18:05.344677 9188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:05.344964 9188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:05.346127 9188 hybrid_clock.cc:648] HybridClock initialized: now 1781983085346104 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:18:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1161 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:18:05 dist-test-slave-mwkb krb5kdc[9201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:18:05 dist-test-slave-mwkb krb5kdc[9201](info): set up 2 sockets
Jun 20 19:18:05 dist-test-slave-mwkb krb5kdc[9201](info): commencing operation
krb5kdc: starting...
W20260620 19:18:06.305306 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:18:06.401273 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33367
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:06.476105 9214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:06.476280 9214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:06.476300 9214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:06.477362 9214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:06.477401 9214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:06.477413 9214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:06.477454 9214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:06.477474 9214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:06.478760 9214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:06.479002 9214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:06.479199 9214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:06.483374 9220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.483412 9222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.483350 9219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:06.483549 9214 server_base.cc:1034] running on GCE node
I20260620 19:18:06.483762 9214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:06.484025 9214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:06.485200 9214 hybrid_clock.cc:648] HybridClock initialized: now 1781983086485179 us; error 35 us; skew 500 ppm
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:06.487391 9214 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:18:06.487783 9214 webserver.cc:460] Webserver started at http://127.1.1.254:46213/ using document root <none> and password file <none>
I20260620 19:18:06.487984 9214 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:06.488049 9214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:06.488180 9214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:06.489019 9214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "81db6a8d2fda4b71ae53bca8b14a8b31"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "773c05d6485a3bec7875ceae6c760e7b"
server_key_iv: "e208240cc2c50b5f0ac600969467d0c1"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.489413 9214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "81db6a8d2fda4b71ae53bca8b14a8b31"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "773c05d6485a3bec7875ceae6c760e7b"
server_key_iv: "e208240cc2c50b5f0ac600969467d0c1"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.490571 9214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:06.491207 9229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.491364 9214 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:06.491449 9214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "81db6a8d2fda4b71ae53bca8b14a8b31"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "773c05d6485a3bec7875ceae6c760e7b"
server_key_iv: "e208240cc2c50b5f0ac600969467d0c1"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.491528 9214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:06.508903 9214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:06.510926 9214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:06.511080 9214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:06.514513 9214 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:33367
I20260620 19:18:06.514540 9281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:33367 every 8 connection(s)
I20260620 19:18:06.514843 9214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:06.515622 9282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:06.517568 9282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Bootstrap starting.
I20260620 19:18:06.518085 9282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:06.518345 9282 log.cc:824] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:06.518920 9282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: No bootstrap required, opened a new log
I20260620 19:18:06.520416 9282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:06.520539 9282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:06.520586 9282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db6a8d2fda4b71ae53bca8b14a8b31, State: Initialized, Role: FOLLOWER
I20260620 19:18:06.520720 9282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:06.520774 9282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:06.520826 9282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:06.520870 9282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:06.521407 9282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:06.521514 9282 leader_election.cc:302] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db6a8d2fda4b71ae53bca8b14a8b31; no voters:
I20260620 19:18:06.521667 9282 leader_election.cc:288] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:18:06.521692 9285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:06.521895 9282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:06.521982 9285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 LEADER]: Becoming Leader. State: Replica: 81db6a8d2fda4b71ae53bca8b14a8b31, State: Running, Role: LEADER
I20260620 19:18:06.522059 9285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:06.522364 9287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db6a8d2fda4b71ae53bca8b14a8b31. Latest consensus state: current_term: 1 leader_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } } }
I20260620 19:18:06.522356 9286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } } }
I20260620 19:18:06.522441 9287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:06.522446 9286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:06.522655 9294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:06.523140 9294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:18:06.524523 9294 catalog_manager.cc:1347] Generated new cluster ID: 800901cd58fd493ea372ed36cf8316fe
I20260620 19:18:06.524580 9294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:18:06.524803 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9214
I20260620 19:18:06.524922 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:18:06.525166 1031 external_mini_cluster.cc:1428] Setting key 5d162ffc627011c6525fe484465c2451
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:06.534271 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.525575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.1.1.254:33367)
0620 19:18:06.525727 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:06.525734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:06.525884 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:18:06.526051 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:06.526056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:06.526095 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:06.526225 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:06.526840 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.527731 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.528472 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.528883 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.529013 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.531212 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:06.531216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:06.531216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:06.531227 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:06.532768 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.533161 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.533164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.533164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.533246 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.533448 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.533449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.533450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.533662 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:06.533811 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:06.533916 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 19:18:06.534061 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 19:18:06.536437 9294 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:18:06.536996 9294 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:06.552214 9294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Generated new TSK 0
I20260620 19:18:06.552359 9294 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:18:06.577899 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:06.655828 9308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:06.656014 9308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:06.656037 9308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:06.657059 9308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:06.657151 9308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:06.657193 9308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:06.658425 9308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:06.658658 9308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:06.658876 9308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:06.663259 9316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.663280 9313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:06.663295 9308 server_base.cc:1034] running on GCE node
W20260620 19:18:06.663285 9314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:06.663647 9308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:06.663842 9308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:06.664996 9308 hybrid_clock.cc:648] HybridClock initialized: now 1781983086664988 us; error 26 us; skew 500 ppm
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:06.667253 9308 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:18:06.667613 9308 webserver.cc:460] Webserver started at http://127.1.1.193:37553/ using document root <none> and password file <none>
I20260620 19:18:06.667812 9308 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:06.667869 9308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:06.667995 9308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:06.668843 9308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "17202aa81b6b4ba289ece499b045045b"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "fb157aece3073a26e8b97e23c1dc31cc"
server_key_iv: "97d98105bb413ac40443cb1170cd2b8a"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.669196 9308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "17202aa81b6b4ba289ece499b045045b"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "fb157aece3073a26e8b97e23c1dc31cc"
server_key_iv: "97d98105bb413ac40443cb1170cd2b8a"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.670274 9308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:18:06.670825 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.670971 9308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.671061 9308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "17202aa81b6b4ba289ece499b045045b"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "fb157aece3073a26e8b97e23c1dc31cc"
server_key_iv: "97d98105bb413ac40443cb1170cd2b8a"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.671134 9308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:06.685324 9308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:06.686934 9308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:06.687081 9308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:06.687275 9308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:06.687559 9308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:06.687595 9308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.687616 9308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:06.687660 9308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.692703 9308 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:34233
I20260620 19:18:06.692770 9436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:34233 every 8 connection(s)
I20260620 19:18:06.693046 9308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:06.701002 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.693786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:50941 (local address 127.1.1.254:33367)
0620 19:18:06.693862 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:06.693864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:06.694040 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:18:06.694564 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:06.694565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:06.694579 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:06.694629 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:06.695539 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.696041 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.696927 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.697208 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.697276 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.698954 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:06.698955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:06.698956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:06.698961 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:06.700050 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.700368 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.700369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.700369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.700410 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.700589 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.700589 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.700590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.700670 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:06.700710 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:06.700894 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 19:18:06.700949 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:18:06.701164 9439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.693758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:50941)
0620 19:18:06.693917 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:06.693963 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:06.694464 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:06.694687 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:06.694691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:06.695262 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:06.695438 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.695442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.696170 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.696171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.696840 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.696841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.697282 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.697284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.697399 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.697738 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:06.697744 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:06.698853 (+ 1109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:06.700117 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.700123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.700124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.700312 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.700464 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.700465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.700465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.700545 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.700716 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:06.700719 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:06.700806 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 19:18:06.700872 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:18:06.700969 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260620 19:18:06.701853 9437 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33367
I20260620 19:18:06.701906 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9308
I20260620 19:18:06.701964 9437 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:06.701998 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:18:06.702138 9437 heartbeater.cc:511] Master 127.1.1.254:33367 requested a full tablet report, sending...
I20260620 19:18:06.702195 1031 external_mini_cluster.cc:1428] Setting key d13f50c6c92d100cc2935409ebf61be6
I20260620 19:18:06.702602 9246 ts_manager.cc:194] Registered new tserver with Master: 17202aa81b6b4ba289ece499b045045b (127.1.1.193:34233)
I20260620 19:18:06.703269 9246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:50941
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:18:06.723172 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:06.793109 9442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:06.793308 9442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:06.793339 9442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:06.794456 9442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:06.794490 9442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:06.794514 9442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:06.795883 9442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:06.796054 9442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:06.796228 9442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:06.800386 9448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.800392 9447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.800397 9450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:06.800454 9442 server_base.cc:1034] running on GCE node
I20260620 19:18:06.800720 9442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:06.800940 9442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:06.802060 9442 hybrid_clock.cc:648] HybridClock initialized: now 1781983086802056 us; error 26 us; skew 500 ppm
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:06.803983 9442 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:18:06.804376 9442 webserver.cc:460] Webserver started at http://127.1.1.194:37783/ using document root <none> and password file <none>
I20260620 19:18:06.804575 9442 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:06.804633 9442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:06.804749 9442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:06.805732 9442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "d7e682ef87414474b150f88e073d4380"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "f3e8e0888e7f81886568c134b0547332"
server_key_iv: "734201744c7678a6d8acd2e294edd37e"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.806066 9442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "d7e682ef87414474b150f88e073d4380"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "f3e8e0888e7f81886568c134b0547332"
server_key_iv: "734201744c7678a6d8acd2e294edd37e"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.807384 9442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:18:06.808094 9457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.808245 9442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:18:06.808331 9442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "d7e682ef87414474b150f88e073d4380"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "f3e8e0888e7f81886568c134b0547332"
server_key_iv: "734201744c7678a6d8acd2e294edd37e"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.808398 9442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:06.834089 9442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:06.835858 9442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:06.836015 9442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:06.836244 9442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:06.836521 9442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:06.836568 9442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.836606 9442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:06.836632 9442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.841936 9442 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:37343
I20260620 19:18:06.841981 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:37343 every 8 connection(s)
I20260620 19:18:06.842242 9442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:18:06.847208 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9442
I20260620 19:18:06.847322 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:18:06.847478 1031 external_mini_cluster.cc:1428] Setting key d9c2caa2a455aba24f42eb1e9a7e5918
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:06.850500 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.842785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:34255)
0620 19:18:06.842940 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:06.842978 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:06.843390 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:06.843613 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:06.843617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:06.844075 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:06.844225 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.844229 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.844822 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.844823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.845658 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.845660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.846044 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.846045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.846175 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.846597 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:06.846604 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:06.847967 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:06.849339 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.849346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.849348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.849607 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.849793 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.849793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.849794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.849874 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.850119 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:06.850122 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:06.850211 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 19:18:06.850280 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:18:06.850375 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260620 19:18:06.850520 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.842818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:34255 (local address 127.1.1.254:33367)
0620 19:18:06.842897 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:06.842899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:06.843048 (+ 149us) server_negotiation.cc:407] Connection header received
0620 19:18:06.843490 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:06.843491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:06.843502 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:06.843545 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:06.844310 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.844740 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.845752 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.845950 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.846036 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.848043 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:06.848044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:06.848045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:06.848052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:06.849251 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.849672 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.849673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.849674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.849731 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.849929 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.849930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.849930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.850054 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:06.850121 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:06.850320 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 19:18:06.850400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:18:06.850853 9571 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33367
I20260620 19:18:06.850951 9571 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:06.851109 9571 heartbeater.cc:511] Master 127.1.1.254:33367 requested a full tablet report, sending...
I20260620 19:18:06.851413 9246 ts_manager.cc:194] Registered new tserver with Master: d7e682ef87414474b150f88e073d4380 (127.1.1.194:37343)
I20260620 19:18:06.851886 9246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:34255
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:18:06.867297 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:06.942126 9576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:06.942292 9576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:06.942314 9576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:06.943423 9576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:06.943463 9576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:06.943487 9576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:06.944875 9576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:06.945153 9576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:06.945364 9576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:06.949962 9582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.949944 9581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:06.949940 9584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:06.950132 9576 server_base.cc:1034] running on GCE node
I20260620 19:18:06.950359 9576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:06.950582 9576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:06.951736 9576 hybrid_clock.cc:648] HybridClock initialized: now 1781983086951728 us; error 26 us; skew 500 ppm
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:06.954133 9576 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:18:06.954521 9576 webserver.cc:460] Webserver started at http://127.1.1.195:36429/ using document root <none> and password file <none>
I20260620 19:18:06.954743 9576 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:06.954809 9576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:06.954953 9576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:06.955880 9576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "b2035b85a03b416993937f62e90c0c0a"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "2b4ebbacbfafd0e929274780416bf931"
server_key_iv: "da9a8ef289de29b7b2fb7f0fda5667c2"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.956238 9576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "b2035b85a03b416993937f62e90c0c0a"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "2b4ebbacbfafd0e929274780416bf931"
server_key_iv: "da9a8ef289de29b7b2fb7f0fda5667c2"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.957495 9576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:18:06.958124 9591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.958281 9576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:18:06.958374 9576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "b2035b85a03b416993937f62e90c0c0a"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "2b4ebbacbfafd0e929274780416bf931"
server_key_iv: "da9a8ef289de29b7b2fb7f0fda5667c2"
server_key_version: "encryptionkey@0"
I20260620 19:18:06.958448 9576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:06.968468 9576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:06.970443 9576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:06.970599 9576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:06.970783 9576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:06.971040 9576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:06.971092 9576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.971127 9576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:06.971153 9576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:06.976378 9576 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34179
I20260620 19:18:06.976445 9704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34179 every 8 connection(s)
I20260620 19:18:06.976715 9576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:18:06.980908 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9576
I20260620 19:18:06.981010 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:18:06.981237 1031 external_mini_cluster.cc:1428] Setting key 016491869585fac3030d6daa6b41d31b
Jun 20 19:18:06 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983086, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:06.985006 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.977386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:50359 (local address 127.1.1.254:33367)
0620 19:18:06.977452 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:06.977454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:06.977592 (+ 138us) server_negotiation.cc:407] Connection header received
0620 19:18:06.977985 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:06.977986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:06.977995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:06.978045 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:06.978801 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.979315 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.980208 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.980385 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.980434 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.982643 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:06.982644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:06.982644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:06.982650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:06.983982 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.984325 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.984326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.984326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.984371 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:06.984560 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:06.984561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:06.984561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:06.984653 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:06.984701 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:06.984892 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:18:06.984947 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:18:06.985234 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:06.977344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:50359)
0620 19:18:06.977480 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:06.977515 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:06.977912 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:06.978102 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:06.978105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:06.978582 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:06.978734 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.978738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.979387 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.979387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.980134 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:06.980136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:06.980452 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:06.980453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:06.980538 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:06.980916 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:06.980926 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:06.982503 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:06.984050 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.984054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.984055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.984264 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.984426 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:06.984427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:06.984427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:06.984509 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:06.984705 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:06.984708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:06.984802 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:18:06.984879 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:18:06.984981 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260620 19:18:06.985636 9705 heartbeater.cc:347] Connected to a master server at 127.1.1.254:33367
I20260620 19:18:06.985714 9705 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:06.985827 9705 heartbeater.cc:511] Master 127.1.1.254:33367 requested a full tablet report, sending...
I20260620 19:18:06.986178 9246 ts_manager.cc:194] Registered new tserver with Master: b2035b85a03b416993937f62e90c0c0a (127.1.1.195:34179)
I20260620 19:18:06.986487 9246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:50359
I20260620 19:18:06.993474 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
Entry for principal oryx/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab.
WARNING: no policy specified for oryx/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
Entry for principal oryx/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab.
WARNING: no policy specified for oryx/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
Entry for principal oryx/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab.
I20260620 19:18:07.050935 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9308
I20260620 19:18:07.055580 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9442
I20260620 19:18:07.059656 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9576
I20260620 19:18:07.063222 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9214
I20260620 19:18:07.067687 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33367
--webserver_interface=127.1.1.254
--webserver_port=46213
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:07.138056 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:07.138281 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:07.138319 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:07.139460 9714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:07.139525 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:07.139550 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:07.139576 9714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:07.139606 9714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:07.141023 9714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=46213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:07.141306 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:07.141521 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:07.145730 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.145741 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.145789 9714 server_base.cc:1034] running on GCE node
W20260620 19:18:07.145737 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.146159 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:07.146453 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:07.147578 9714 hybrid_clock.cc:648] HybridClock initialized: now 1781983087147569 us; error 24 us; skew 500 ppm
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:07.149595 9714 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:18:07.149989 9714 webserver.cc:460] Webserver started at http://127.1.1.254:46213/ using document root <none> and password file <none>
I20260620 19:18:07.150182 9714 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:07.150228 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:07.151319 9714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:07.151844 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.151999 9714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.152081 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "81db6a8d2fda4b71ae53bca8b14a8b31"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "773c05d6485a3bec7875ceae6c760e7b"
server_key_iv: "e208240cc2c50b5f0ac600969467d0c1"
server_key_version: "encryptionkey@0"
I20260620 19:18:07.152346 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:07.167595 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:07.168964 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:07.169134 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:07.172456 9714 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:33367
I20260620 19:18:07.172483 9781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:33367 every 8 connection(s)
I20260620 19:18:07.172816 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:07.174045 9782 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:18:07.174624 9782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Bootstrap starting.
I20260620 19:18:07.181089 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9714
I20260620 19:18:07.181360 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:34233
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:33367
--webserver_port=37553
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:18:07.182201 9782 log.cc:824] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:07.183544 9782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:18:07.183772 9782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Bootstrap complete.
I20260620 19:18:07.185847 9782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:07.186076 9782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81db6a8d2fda4b71ae53bca8b14a8b31, State: Initialized, Role: FOLLOWER
I20260620 19:18:07.186285 9782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:07.186365 9782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:07.186427 9782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:07.186470 9782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:18:07.187286 9782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:07.187392 9782 leader_election.cc:302] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81db6a8d2fda4b71ae53bca8b14a8b31; no voters:
I20260620 19:18:07.187600 9782 leader_election.cc:288] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:18:07.187628 9786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:18:07.187724 9786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [term 2 LEADER]: Becoming Leader. State: Replica: 81db6a8d2fda4b71ae53bca8b14a8b31, State: Running, Role: LEADER
I20260620 19:18:07.187798 9786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } }
I20260620 19:18:07.187822 9782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:07.188046 9786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81db6a8d2fda4b71ae53bca8b14a8b31. Latest consensus state: current_term: 2 leader_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } } }
I20260620 19:18:07.188050 9787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81db6a8d2fda4b71ae53bca8b14a8b31" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 33367 } } }
I20260620 19:18:07.188262 9787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:07.188114 9786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:07.189575 9801 catalog_manager.cc:1259] Loaded cluster ID: 800901cd58fd493ea372ed36cf8316fe
I20260620 19:18:07.190336 9801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: loading cluster ID for follower catalog manager: success
I20260620 19:18:07.191484 9801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: acquiring CA information for follower catalog manager: success
I20260620 19:18:07.191771 9801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:18:07.191835 9795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:07.191974 9795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:18:07.192037 9795 catalog_manager.cc:1259] Loaded cluster ID: 800901cd58fd493ea372ed36cf8316fe
I20260620 19:18:07.192066 9795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:18:07.192211 9795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:07.192267 9795 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 81db6a8d2fda4b71ae53bca8b14a8b31: Loaded TSK: 0
I20260620 19:18:07.192375 9795 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:18:07.257148 9785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:07.257301 9785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:07.257320 9785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:07.258277 9785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:07.258311 9785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:07.258333 9785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:07.259603 9785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.193.keytab
--principal=oryx/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37553
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:07.259797 9785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:07.260013 9785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:18:07.264994 9785 server_base.cc:1034] running on GCE node
W20260620 19:18:07.264971 9808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.264987 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.265118 9811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.265462 9785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:07.265686 9785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:07.266836 9785 hybrid_clock.cc:648] HybridClock initialized: now 1781983087266826 us; error 28 us; skew 500 ppm
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983087, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:07.268927 9785 init.cc:377] Logged in from keytab as oryx/127.1.1.193@KRBTEST.COM (short username oryx)
I20260620 19:18:07.269346 9785 webserver.cc:460] Webserver started at http://127.1.1.193:37553/ using document root <none> and password file <none>
I20260620 19:18:07.269562 9785 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:07.269619 9785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:07.270797 9785 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:18:07.271378 9818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.271513 9785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:18:07.271603 9785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "17202aa81b6b4ba289ece499b045045b"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "fb157aece3073a26e8b97e23c1dc31cc"
server_key_iv: "97d98105bb413ac40443cb1170cd2b8a"
server_key_version: "encryptionkey@0"
I20260620 19:18:07.271863 9785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:07.282447 9785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:07.284194 9785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:07.284312 9785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:07.284518 9785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:07.284803 9785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:07.284842 9785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.284859 9785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:07.284870 9785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.291280 9785 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:34233
I20260620 19:18:07.291380 9931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:34233 every 8 connection(s)
I20260620 19:18:07.291616 9785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:18:07.295833 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9785
I20260620 19:18:07.296125 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:37343
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:33367
--webserver_port=37783
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.299777 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.292213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:35579)
0620 19:18:07.292351 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.292383 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.292717 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.293134 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.293137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.293603 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.293750 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.293754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.295218 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.295219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.296047 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.296050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.296714 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.296715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.296821 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.297263 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.297272 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.299620 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
W20260620 19:18:07.299961 9932 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:33367 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.1.254:33367: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
W20260620 19:18:07.300278 9935 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.292274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:35579 (local address 127.1.1.254:33367)
0620 19:18:07.292425 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.292432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.292460 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:18:07.292890 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.292896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.292941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.293085 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.293859 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.295127 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.296127 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.296621 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.296739 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.299756 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:35579: BlockingRecv error: recv got EOF from 127.1.1.193:35579 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.305023 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.300131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:47993)
0620 19:18:07.300384 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.300393 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.300427 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.300643 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.300644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.300771 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.300811 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.300812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.301662 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.301663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.302397 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.302398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.302853 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.302853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.302905 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.303277 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.303281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.304941 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":93,"threads_started":1}
W20260620 19:18:07.305334 9935 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.300292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:47993 (local address 127.1.1.254:33367)
0620 19:18:07.300511 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.300524 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.300532 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:07.300551 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.300551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.300563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.300625 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.300884 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.301593 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.302473 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.302780 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.302849 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.305261 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:47993: BlockingRecv error: recv got EOF from 127.1.1.193:47993 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":53,"threads_started":1}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.309415 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.305239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:39609)
0620 19:18:07.305333 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.305342 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.305373 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.305619 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.305620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.305772 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.305808 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.305809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.306650 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.306651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.307362 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.307363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.307653 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.307653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.307697 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.307980 (+ 283us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.307984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.309354 (+ 1370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 19:18:07.309470 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.305303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:39609 (local address 127.1.1.254:33367)
0620 19:18:07.305436 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.305438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.305444 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:07.305461 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.305462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.305470 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.305584 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.305899 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.306572 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.307429 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.307596 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.307635 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.309412 (+ 1777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:39609: BlockingRecv error: recv got EOF from 127.1.1.193:39609 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260620 19:18:07.309504 9932 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:18:07.370390 9936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:07.370566 9936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:07.370596 9936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:07.371554 9936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:07.371608 9936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:07.371646 9936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:07.372828 9936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.194.keytab
--principal=oryx/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:37343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=37783
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:07.373045 9936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:07.373242 9936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:07.377691 9943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.377671 9944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.377727 9936 server_base.cc:1034] running on GCE node
W20260620 19:18:07.377709 9946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.378015 9936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:07.378237 9936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:07.379371 9936 hybrid_clock.cc:648] HybridClock initialized: now 1781983087379365 us; error 34 us; skew 500 ppm
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983087, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:07.381603 9936 init.cc:377] Logged in from keytab as oryx/127.1.1.194@KRBTEST.COM (short username oryx)
I20260620 19:18:07.381990 9936 webserver.cc:460] Webserver started at http://127.1.1.194:37783/ using document root <none> and password file <none>
I20260620 19:18:07.382184 9936 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:07.382256 9936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:07.383316 9936 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:07.383791 9953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.383934 9936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.384016 9936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "d7e682ef87414474b150f88e073d4380"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "f3e8e0888e7f81886568c134b0547332"
server_key_iv: "734201744c7678a6d8acd2e294edd37e"
server_key_version: "encryptionkey@0"
I20260620 19:18:07.384277 9936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:07.414624 9936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:07.416162 9936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:07.416325 9936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:07.416520 9936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:07.416780 9936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:07.416826 9936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.416858 9936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:07.416880 9936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.422180 9936 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:37343
I20260620 19:18:07.422217 10066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:37343 every 8 connection(s)
I20260620 19:18:07.422492 9936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:18:07.428963 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.423056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:46863 (local address 127.1.1.254:33367)
0620 19:18:07.423115 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.423117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.423307 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:18:07.423722 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.423723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.423735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.423791 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.424619 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.425213 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.426154 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.426368 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.426442 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.428894 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:46863: BlockingRecv error: recv got EOF from 127.1.1.194:46863 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:18:07.428992 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.423052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:46863)
0620 19:18:07.423210 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.423236 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.423632 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.423850 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.423853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.424343 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.424500 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.424503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.425297 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.425298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.426062 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.426064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.426461 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.426461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.426576 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.426956 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.426963 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.428820 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
W20260620 19:18:07.429185 10067 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:33367 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.1.254:33367: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
I20260620 19:18:07.430992 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 9936
I20260620 19:18:07.431236 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:34179
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:33367
--webserver_port=36429
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.433169 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.429383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:59971)
0620 19:18:07.429427 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.429437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.429472 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.429595 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.429595 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.429716 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.429750 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.429751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.430334 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.430335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.430775 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.430777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.431124 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.431125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.431182 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.431511 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.431514 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.433059 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:18:07.433259 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.429446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:59971 (local address 127.1.1.254:33367)
0620 19:18:07.429486 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.429488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.429496 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:07.429517 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.429518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.429528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.429593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.429820 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.430265 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.430873 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.431064 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.431115 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.433196 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:59971: BlockingRecv error: recv got EOF from 127.1.1.194:59971 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.437773 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.433356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:42711)
0620 19:18:07.433467 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.433476 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.433540 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.433645 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.433646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.433824 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.433872 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.433874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.434395 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.434396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.435360 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.435362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.435693 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.435694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.435771 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.436123 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.436129 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.437728 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":74,"threads_started":1}
W20260620 19:18:07.437889 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.433385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:42711 (local address 127.1.1.254:33367)
0620 19:18:07.433409 (+ 24us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.433411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.433527 (+ 116us) server_negotiation.cc:407] Connection header received
0620 19:18:07.433572 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.433572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.433579 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.433619 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.433945 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.434328 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.435438 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.435602 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.435650 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.437821 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:42711: BlockingRecv error: recv got EOF from 127.1.1.194:42711 (error 108)
Metrics: {"server-negotiator.queue_time_us":11}
W20260620 19:18:07.437942 10067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:18:07.503451 10070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:07.503611 10070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:07.503633 10070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:07.504565 10070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:07.504594 10070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:07.504617 10070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:07.505970 10070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/krb5kdc/oryx_127.1.1.195.keytab
--principal=oryx/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:34179
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=36429
--tserver_master_addrs=127.1.1.254:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:07.506131 10070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:07.506301 10070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:07.510435 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.510426 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:07.510416 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:07.510532 10070 server_base.cc:1034] running on GCE node
I20260620 19:18:07.510752 10070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:07.510974 10070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:07.512128 10070 hybrid_clock.cc:648] HybridClock initialized: now 1781983087512129 us; error 40 us; skew 500 ppm
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983087, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:07.514066 10070 init.cc:377] Logged in from keytab as oryx/127.1.1.195@KRBTEST.COM (short username oryx)
I20260620 19:18:07.514436 10070 webserver.cc:460] Webserver started at http://127.1.1.195:36429/ using document root <none> and password file <none>
I20260620 19:18:07.514631 10070 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:07.514696 10070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:07.515866 10070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:07.516373 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.516526 10070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:07.516610 10070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "b2035b85a03b416993937f62e90c0c0a"
format_stamp: "Formatted at 2026-06-20 19:18:06 on dist-test-slave-mwkb"
server_key: "2b4ebbacbfafd0e929274780416bf931"
server_key_iv: "da9a8ef289de29b7b2fb7f0fda5667c2"
server_key_version: "encryptionkey@0"
I20260620 19:18:07.516942 10070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:07.538154 10070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:07.539862 10070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:07.540019 10070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:07.540267 10070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:07.540539 10070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:07.540584 10070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.540620 10070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:07.540644 10070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:07.546156 10070 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34179
I20260620 19:18:07.546202 10199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34179 every 8 connection(s)
I20260620 19:18:07.546511 10070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:18:07.552695 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.547136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:60939 (local address 127.1.1.254:33367)
0620 19:18:07.547173 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.547175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.547347 (+ 172us) server_negotiation.cc:407] Connection header received
0620 19:18:07.547767 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.547767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.547775 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.547838 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.548665 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.549147 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.550003 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.550162 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.550207 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.552653 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:60939: BlockingRecv error: recv got EOF from 127.1.1.195:60939 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:18:07.552697 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.547076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:60939)
0620 19:18:07.547242 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.547279 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.547693 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.547910 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.547914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.548443 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.548599 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.548603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.549226 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.549227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.549933 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.549934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.550232 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.550233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.550314 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.550701 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.550707 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.552579 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260620 19:18:07.552938 10200 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:33367 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.1.254:33367: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.556154 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10070
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.556816 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.553147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:38211)
0620 19:18:07.553188 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.553197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.553232 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.553407 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.553407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.553501 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.553535 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.553536 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.554075 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.554076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.554544 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.554545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.554828 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.554829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.554872 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.555212 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.555215 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.556750 (+ 1535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:18:07.556862 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.553269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:38211 (local address 127.1.1.254:33367)
0620 19:18:07.553314 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.553315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.553321 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:07.553338 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.553339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.553352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.553393 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.553588 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.554007 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.554606 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.554768 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.554815 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.556808 (+ 1993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:38211: BlockingRecv error: recv got EOF from 127.1.1.195:38211 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:07 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:07.561945 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.557261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:37583)
0620 19:18:07.557344 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:07.557352 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:07.557390 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:07.557961 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:07.557961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:07.558063 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:07.558083 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.558084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.558829 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.558830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.559535 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:07.559538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.559973 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.559973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:07.560007 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.560332 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:07.560336 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:07.561914 (+ 1578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 19:18:07.562057 9938 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:07.557310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:37583 (local address 127.1.1.254:33367)
0620 19:18:07.557370 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.557373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.557385 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:18:07.557426 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.557427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.557436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.557506 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.558146 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.558764 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.559608 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.559884 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.559960 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.562005 (+ 2045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:37583: BlockingRecv error: recv got EOF from 127.1.1.195:37583 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 19:18:07.562084 10200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:18:07.563248 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:07.556509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.1.1.254:33367)
0620 19:18:07.556562 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:07.556563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:07.556691 (+ 128us) server_negotiation.cc:407] Connection header received
0620 19:18:07.556761 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:07.556762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:07.556773 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:07.556816 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:07.557522 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.557899 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.558810 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:07.559112 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:07.559176 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:07.560356 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:07.560361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:07.560361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:07.560370 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:07.562112 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:07.562411 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:07.562413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:07.562414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:07.562496 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:07.562679 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:07.562679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:07.562680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:07.562859 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:07.562990 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:07.563091 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 19:18:07.563185 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:08.314208 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:08.309818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:60971)
0620 19:18:08.309929 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:08.309946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:08.309988 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:08.310154 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:08.310155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:08.310292 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:08.310336 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.310338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.310968 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.310969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.311448 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.311449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.311910 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.311911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.311968 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.312456 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:08.312461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:08.314152 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":74,"threads_started":1}
W20260620 19:18:08.314304 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:08.309899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60971 (local address 127.1.1.254:33367)
0620 19:18:08.310016 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:08.310017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:08.310024 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:18:08.310041 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:08.310042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:08.310050 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:08.310105 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:08.310406 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.310900 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.311548 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.311843 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.311901 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.314237 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:60971: BlockingRecv error: recv got EOF from 127.1.1.193:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":72,"threads_started":1}
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:08.442343 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:08.438267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:56807)
0620 19:18:08.438398 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:08.438408 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:08.438458 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:08.438579 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:08.438580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:08.438725 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:08.438771 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.438772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.439410 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.439410 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.440023 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.440025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.440360 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.440361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.440427 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.440788 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:08.440793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:08.442260 (+ 1467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
W20260620 19:18:08.442386 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:08.438337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:56807 (local address 127.1.1.254:33367)
0620 19:18:08.438403 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:08.438409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:08.438448 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:18:08.438509 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:08.438510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:08.438519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:08.438559 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:08.438835 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.439348 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.440120 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.440296 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.440361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.442330 (+ 1969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:56807: BlockingRecv error: recv got EOF from 127.1.1.194:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:08 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:08.566592 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:08.562420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:34707)
0620 19:18:08.562569 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:08.562587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:08.562636 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:08.562763 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:08.562763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:08.562894 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:08.562945 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.562946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.563497 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.563498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.563973 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:08.563974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.564253 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.564254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:08.564298 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.564770 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:08.564774 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:08.566482 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
W20260620 19:18:08.566622 10205 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:08.562592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:34707 (local address 127.1.1.254:33367)
0620 19:18:08.562654 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:08.562655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:08.562674 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:18:08.562692 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:08.562693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:08.562702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:08.562742 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:08.563016 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.563424 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.564040 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:08.564198 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:08.564243 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:08.566548 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:34707: BlockingRecv error: recv got EOF from 127.1.1.195:34707 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:09.323258 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:09.314643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:58657)
0620 19:18:09.314858 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:09.314868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:09.314943 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:09.315127 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:09.315128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:09.315342 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:09.315399 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.315402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.316034 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.316034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.316551 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.316553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.316853 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.316854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.316889 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.321703 (+ 4814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:09.321707 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:09.323211 (+ 1504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":126,"threads_started":1}
W20260620 19:18:09.323392 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:09.314809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:58657 (local address 127.1.1.254:33367)
0620 19:18:09.314934 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:09.314936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:09.314944 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:09.315032 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:09.315033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:09.315045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:09.315114 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:09.315473 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.315967 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.316615 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.316798 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.316843 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.323305 (+ 6462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:58657: BlockingRecv error: recv got EOF from 127.1.1.193:58657 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:09.446748 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:09.442758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:56591)
0620 19:18:09.442914 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:09.442923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:09.442970 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:09.443097 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:09.443097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:09.443232 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:09.443286 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.443287 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.443865 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.443865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.444323 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.444324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.444603 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.444604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.444636 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.445009 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:09.445013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:09.446697 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260620 19:18:09.446897 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:09.442807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:56591 (local address 127.1.1.254:33367)
0620 19:18:09.442866 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:09.442868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:09.442974 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:18:09.443030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:09.443031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:09.443040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:09.443076 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:09.443338 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.443809 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.444384 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.444547 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.444591 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.446818 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:56591: BlockingRecv error: recv got EOF from 127.1.1.194:56591 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:09 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:09.571234 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:09.567133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:53997)
0620 19:18:09.567249 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:09.567259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:09.567325 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:09.567484 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:09.567485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:09.567639 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:09.567682 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.567683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.568291 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.568293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.568755 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:09.568757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.569192 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.569193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:09.569260 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.569685 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:09.569690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:09.571189 (+ 1499us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
W20260620 19:18:09.571374 10209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:09.567257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:53997 (local address 127.1.1.254:33367)
0620 19:18:09.567314 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:09.567316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:09.567325 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:09.567383 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:09.567384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:09.567397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:09.567477 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:09.567752 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.568209 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.568829 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:09.569039 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:09.569106 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:09.571301 (+ 2195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:53997: BlockingRecv error: recv got EOF from 127.1.1.195:53997 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:10.328325 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:10.323635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:51479)
0620 19:18:10.323791 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:10.323800 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:10.323842 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:10.323985 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:10.323986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:10.324124 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:10.324169 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.324170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.324878 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.324879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.325502 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.325505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.325905 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.325906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.325987 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.326493 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:10.326498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:10.328206 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
W20260620 19:18:10.328382 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:10.323691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:51479 (local address 127.1.1.254:33367)
0620 19:18:10.323821 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:10.323823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:10.323832 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:10.323898 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:10.323899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:10.323910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:10.323968 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:10.324225 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.324682 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.325585 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.325807 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.325903 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.328307 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:51479: BlockingRecv error: recv got EOF from 127.1.1.193:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:10.451095 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:10.447140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:56193)
0620 19:18:10.447294 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:10.447306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:10.447373 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:10.447482 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:10.447482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:10.447630 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:10.447683 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.447685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.448272 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.448273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.448768 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.448769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.449059 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.449060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.449150 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.449554 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:10.449558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:10.451033 (+ 1475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
W20260620 19:18:10.451174 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:10.447197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:56193 (local address 127.1.1.254:33367)
0620 19:18:10.447249 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:10.447252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:10.447356 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:18:10.447413 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:10.447414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:10.447423 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:10.447466 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:10.447742 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.448213 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.448838 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.449004 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.449050 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.451114 (+ 2064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:56193: BlockingRecv error: recv got EOF from 127.1.1.194:56193 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:10 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:10.575739 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:10.571635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:40713)
0620 19:18:10.571767 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:10.571777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:10.571829 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:10.571953 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:10.571954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:10.572093 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:10.572136 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.572137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.572755 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.572756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.573239 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:10.573241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.573582 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.573583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:10.573637 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.574045 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:10.574049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:10.575677 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
W20260620 19:18:10.575861 10213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:10.571688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40713 (local address 127.1.1.254:33367)
0620 19:18:10.571731 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:10.571734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:10.571815 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:18:10.571865 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:10.571866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:10.571876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:10.571936 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:10.572236 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.572668 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.573315 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:10.573507 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:10.573582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:10.575766 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:40713: BlockingRecv error: recv got EOF from 127.1.1.195:40713 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:11.333140 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:11.328883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:51171)
0620 19:18:11.329042 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:11.329053 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:11.329158 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:11.329283 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:11.329284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:11.329439 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:11.329491 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.329492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.330075 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.330076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.330686 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.330688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.331046 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.331047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.331086 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.331515 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:11.331519 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:11.333030 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
W20260620 19:18:11.333221 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:11.328980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:51171 (local address 127.1.1.254:33367)
0620 19:18:11.329110 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:11.329112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:11.329135 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:18:11.329199 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:11.329200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:11.329210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:11.329264 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:11.329554 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.330019 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.330738 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.330977 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.331034 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.333160 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:51171: BlockingRecv error: recv got EOF from 127.1.1.193:51171 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:11.457183 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:11.451644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:52567)
0620 19:18:11.451788 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:11.451800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:11.451866 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:11.452063 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:11.452064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:11.452306 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:11.452364 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.452366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.453326 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.453327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.453988 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.453991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.454476 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.454477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.454522 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.454981 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:11.454986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:11.457094 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
W20260620 19:18:11.457244 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:11.451744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52567 (local address 127.1.1.254:33367)
0620 19:18:11.451792 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:11.451794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:11.451847 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:18:11.451943 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:11.451944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:11.451956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:11.452001 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:11.452455 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.453235 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.454076 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.454386 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.454462 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.457185 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:52567: BlockingRecv error: recv got EOF from 127.1.1.194:52567 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:11 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:11.580245 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:11.576199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:40893)
0620 19:18:11.576320 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:11.576330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:11.576387 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:11.576506 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:11.576506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:11.576650 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:11.576696 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.576698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.577368 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.577369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.577844 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:11.577845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.578130 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.578131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:11.578163 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.578569 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:11.578573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:11.580194 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
W20260620 19:18:11.580336 10217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:11.576232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40893 (local address 127.1.1.254:33367)
0620 19:18:11.576277 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:11.576280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:11.576391 (+ 111us) server_negotiation.cc:407] Connection header received
0620 19:18:11.576426 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:11.576426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:11.576437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:11.576485 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:11.576754 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.577310 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.577920 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:11.578071 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:11.578118 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:11.580273 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:40893: BlockingRecv error: recv got EOF from 127.1.1.195:40893 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:12.337962 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:12.333604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:33951)
0620 19:18:12.333761 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:12.333770 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:12.333819 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:12.333944 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:12.333945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:12.334088 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:12.334134 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.334135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.334827 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.334829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.335365 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.335367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.335784 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.335785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.335851 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.336270 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:12.336275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:12.337900 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
W20260620 19:18:12.338109 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:12.333690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:33951 (local address 127.1.1.254:33367)
0620 19:18:12.333805 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:12.333806 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:12.333814 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:12.333864 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:12.333864 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:12.333873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:12.333927 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:12.334196 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.334749 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.335499 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.335698 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.335782 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.338026 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:33951: BlockingRecv error: recv got EOF from 127.1.1.193:33951 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:12.461844 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:12.457712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:44559)
0620 19:18:12.457858 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:12.457867 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:12.457909 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:12.458049 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:12.458050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:12.458181 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:12.458223 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.458224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.458805 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.458806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.459345 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.459347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.459650 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.459650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.459687 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.460171 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:12.460175 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:12.461780 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260620 19:18:12.461953 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:12.457863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:44559 (local address 127.1.1.254:33367)
0620 19:18:12.457943 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:12.457947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:12.457957 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:18:12.457976 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:12.457977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:12.457986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:12.458033 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:12.458285 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.458747 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.459418 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.459590 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.459637 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.461856 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:44559: BlockingRecv error: recv got EOF from 127.1.1.194:44559 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:12 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:12.584554 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:12.580648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:49769)
0620 19:18:12.580769 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:12.580780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:12.580839 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:12.580970 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:12.580970 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:12.581130 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:12.581171 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.581173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.581729 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.581730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.582196 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:12.582197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.582482 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.582483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:12.582535 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.582993 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:12.582998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:12.584508 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
W20260620 19:18:12.584693 10221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:12.580731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:49769 (local address 127.1.1.254:33367)
0620 19:18:12.580787 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:12.580789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:12.580821 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:18:12.580890 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:12.580891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:12.580901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:12.580949 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:12.581223 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.581649 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.582260 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:12.582426 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:12.582472 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:12.584621 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:49769: BlockingRecv error: recv got EOF from 127.1.1.195:49769 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:13.342592 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:13.338538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:55159)
0620 19:18:13.338662 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:13.338672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:13.338725 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:13.338837 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:13.338837 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:13.339016 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:13.339062 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.339064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.339671 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.339671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.340186 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.340187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.340466 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.340467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.340520 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.340960 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:13.340964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:13.342510 (+ 1546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260620 19:18:13.342644 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:13.338606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:55159 (local address 127.1.1.254:33367)
0620 19:18:13.338729 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:13.338730 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:13.338757 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:18:13.338775 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:13.338775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:13.338783 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:13.338835 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:13.339125 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.339607 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.340245 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.340399 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.340464 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.342581 (+ 2117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:55159: BlockingRecv error: recv got EOF from 127.1.1.193:55159 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:13.466112 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:13.462249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:46973)
0620 19:18:13.462409 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:13.462420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:13.462475 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:13.462596 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:13.462597 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:13.462732 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:13.462773 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.462775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.463302 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.463302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.463734 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.463735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.464022 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.464023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.464055 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.464424 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:13.464428 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:13.466057 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
W20260620 19:18:13.466220 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:13.462432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:46973 (local address 127.1.1.254:33367)
0620 19:18:13.462485 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:13.462487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:13.462511 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:18:13.462531 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:13.462531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:13.462540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:13.462574 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:13.462827 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.463244 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.463798 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.463967 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.464013 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.466149 (+ 2136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:46973: BlockingRecv error: recv got EOF from 127.1.1.194:46973 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:13 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:13.589592 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:13.585017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:58097)
0620 19:18:13.585191 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:13.585202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:13.585273 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:13.585417 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:13.585418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:13.585662 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:13.585723 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.585725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.586367 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.586367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.586976 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:13.586978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.587281 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.587281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:13.587316 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.587761 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:13.587766 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:13.589507 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":109,"threads_started":1}
W20260620 19:18:13.589625 10225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:13.585103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:58097 (local address 127.1.1.254:33367)
0620 19:18:13.585274 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:13.585276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:13.585281 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:18:13.585352 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:13.585352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:13.585366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:13.585411 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:13.585790 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.586266 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.587041 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:13.587199 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:13.587256 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:13.589570 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:58097: BlockingRecv error: recv got EOF from 127.1.1.195:58097 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:14.351632 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:14.343104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:43253)
0620 19:18:14.343266 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:14.343277 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:14.343334 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:14.343459 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:14.343460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:14.343599 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:14.343643 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.343644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.344389 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.344391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.345050 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.345052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.345406 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.345406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.345442 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.350053 (+ 4611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:14.350064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:14.351581 (+ 1517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
W20260620 19:18:14.351733 10229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:14.343176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:43253 (local address 127.1.1.254:33367)
0620 19:18:14.343273 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:14.343274 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:14.343320 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:18:14.343380 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:14.343380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:14.343389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:14.343439 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:14.343700 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.344261 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.345150 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.345346 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.345395 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.351663 (+ 6268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:43253: BlockingRecv error: recv got EOF from 127.1.1.193:43253 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":42,"threads_started":1}
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:14.470309 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:14.466578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:54065)
0620 19:18:14.466705 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:14.466715 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:14.466774 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:14.466900 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:14.466900 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:14.467029 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:14.467068 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.467069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.467593 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.467593 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.468059 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.468060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.468379 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.468380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.468415 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.468785 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:14.468789 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:14.470259 (+ 1470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260620 19:18:14.470398 10229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:14.466652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:54065 (local address 127.1.1.254:33367)
0620 19:18:14.466698 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:14.466702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:14.466764 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:18:14.466835 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:14.466835 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:14.466844 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:14.466888 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:14.467122 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.467536 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.468127 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.468319 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.468369 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.470339 (+ 1970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:54065: BlockingRecv error: recv got EOF from 127.1.1.194:54065 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:14 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:14.593777 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:14.590002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:38509)
0620 19:18:14.590137 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:14.590147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:14.590228 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:14.590369 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:14.590369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:14.590505 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:14.590546 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.590548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.591074 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.591075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.591519 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:14.591520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.591781 (+ 261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.591782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:14.591814 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.592169 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:14.592173 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:14.593731 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260620 19:18:14.593879 10229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:14.590032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:38509 (local address 127.1.1.254:33367)
0620 19:18:14.590100 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:14.590102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:14.590229 (+ 127us) server_negotiation.cc:407] Connection header received
0620 19:18:14.590299 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:14.590299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:14.590313 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:14.590358 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:14.590598 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.591008 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.591585 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:14.591729 (+ 144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:14.591770 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:14.593809 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:38509: BlockingRecv error: recv got EOF from 127.1.1.195:38509 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:15.356600 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:15.352057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:50789)
0620 19:18:15.352173 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:15.352182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:15.352242 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:15.352482 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:15.352483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:15.352629 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:15.352675 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.352677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.353301 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.353302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.353804 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.353805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.354123 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.354123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.354160 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.354655 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:15.354660 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:15.356508 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260620 19:18:15.356648 10233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:15.352246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:50789 (local address 127.1.1.254:33367)
0620 19:18:15.352372 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:15.352374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:15.352380 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:15.352401 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:15.352401 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:15.352409 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:15.352465 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:15.352737 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.353244 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.353875 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.354063 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.354111 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.356580 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:50789: BlockingRecv error: recv got EOF from 127.1.1.193:50789 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:15.474967 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:15.470733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:55803)
0620 19:18:15.470911 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:15.470927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:15.470982 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:15.471093 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:15.471094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:15.471246 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:15.471291 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.471292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.472086 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.472087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.472559 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.472560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.472824 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.472825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.472862 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.473320 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:15.473324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:15.474863 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":57,"threads_started":1}
W20260620 19:18:15.475013 10233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:15.470779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:55803 (local address 127.1.1.254:33367)
0620 19:18:15.470918 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:15.470920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:15.470967 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:18:15.471018 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:15.471019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:15.471028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:15.471068 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:15.471381 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.472024 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.472614 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.472766 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.472808 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.474946 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:55803: BlockingRecv error: recv got EOF from 127.1.1.194:55803 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:15 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:15.598745 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:15.594259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:50737)
0620 19:18:15.594383 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:15.594392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:15.594440 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:15.594557 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:15.594558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:15.594697 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:15.594738 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.594739 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.595324 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.595325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.596211 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:15.596213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.596519 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.596520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:15.596556 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.597036 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:15.597041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:15.598662 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
W20260620 19:18:15.598798 10233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:15.594335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:50737 (local address 127.1.1.254:33367)
0620 19:18:15.594396 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:15.594400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:15.594430 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:18:15.594493 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:15.594493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:15.594504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:15.594541 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:15.594796 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.595240 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.596278 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:15.596457 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:15.596507 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:15.598739 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:50737: BlockingRecv error: recv got EOF from 127.1.1.195:50737 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:16.361018 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:16.357204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:36231)
0620 19:18:16.357327 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:16.357338 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:16.357398 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:16.357496 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:16.357496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:16.357635 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:16.357678 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.357680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.358213 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.358214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.358666 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.358667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.358979 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.358980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.359015 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.359412 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:16.359416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:16.360965 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
W20260620 19:18:16.361155 10237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:16.357238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:36231 (local address 127.1.1.254:33367)
0620 19:18:16.357353 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:16.357355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:16.357383 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:18:16.357436 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:16.357437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:16.357446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:16.357493 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:16.357732 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.358157 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.358725 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.358916 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.358966 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.361049 (+ 2083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:36231: BlockingRecv error: recv got EOF from 127.1.1.193:36231 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:16.479848 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:16.475380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:37577)
0620 19:18:16.475521 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:16.475533 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:16.475618 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:16.475777 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:16.475778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:16.475950 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:16.475991 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.475993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.476650 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.476651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.477265 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.477267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.477595 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.477596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.477652 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.478088 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:16.478093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:16.479755 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
W20260620 19:18:16.479888 10237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:16.475445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:37577 (local address 127.1.1.254:33367)
0620 19:18:16.475512 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:16.475515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:16.475599 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:18:16.475678 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:16.475679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:16.475691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:16.475771 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:16.476076 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.476567 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.477346 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.477516 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.477594 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.479828 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:37577: BlockingRecv error: recv got EOF from 127.1.1.194:37577 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:16 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:16.603081 10239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:16.599165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:54125)
0620 19:18:16.599282 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:16.599292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:16.599354 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:16.599474 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:16.599474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:16.599614 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:16.599659 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.599661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.600243 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.600244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.600709 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:16.600710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.600984 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.600985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:16.601019 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.601421 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:16.601425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:16.603031 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
W20260620 19:18:16.603178 10237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:16.599217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:54125 (local address 127.1.1.254:33367)
0620 19:18:16.599257 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:16.599259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:16.599341 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:18:16.599411 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:16.599411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:16.599421 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:16.599459 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:16.599720 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.600187 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.600772 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:16.600930 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:16.600973 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:16.603113 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:54125: BlockingRecv error: recv got EOF from 127.1.1.195:54125 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:17.365350 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:17.361408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:45381)
0620 19:18:17.361584 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:17.361613 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:17.361676 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:17.361808 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:17.361809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:17.361957 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:17.362005 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.362013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.362621 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.362622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.363111 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.363112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.363372 (+ 260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.363373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.363407 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.363803 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:17.363807 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:17.365301 (+ 1494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":134,"threads_started":1}
W20260620 19:18:17.365438 10241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:17.361461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:45381 (local address 127.1.1.254:33367)
0620 19:18:17.361587 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:17.361608 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:17.361661 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:18:17.361719 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:17.361719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:17.361729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:17.361786 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:17.362070 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.362556 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.363167 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.363320 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.363361 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.365374 (+ 2013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:45381: BlockingRecv error: recv got EOF from 127.1.1.193:45381 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:17.484468 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:17.480343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:38137)
0620 19:18:17.480496 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:17.480506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:17.480568 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:17.480713 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:17.480714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:17.480872 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:17.480913 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.480915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.481567 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.481568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.482089 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.482091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.482396 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.482397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.482447 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.482937 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:17.482941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:17.484387 (+ 1446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
W20260620 19:18:17.484500 10241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:17.480420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:38137 (local address 127.1.1.254:33367)
0620 19:18:17.480479 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:17.480481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:17.480559 (+ 78us) server_negotiation.cc:407] Connection header received
0620 19:18:17.480626 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:17.480627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:17.480637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:17.480696 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:17.480979 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.481485 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.482160 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.482333 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.482396 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.484449 (+ 2053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:38137: BlockingRecv error: recv got EOF from 127.1.1.194:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:17 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:17.607659 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:17.603588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:60871)
0620 19:18:17.603790 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:17.603800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:17.603871 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:17.604044 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:17.604045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:17.604211 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:17.604256 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.604257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.604795 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.604796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.605246 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:17.605247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.605510 (+ 263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.605510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:17.605544 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.605946 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:17.605951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:17.607606 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
W20260620 19:18:17.607815 10241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:17.603697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:60871 (local address 127.1.1.254:33367)
0620 19:18:17.603740 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:17.603742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:17.603863 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:18:17.603954 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:17.603955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:17.603969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:17.604021 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:17.604328 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.604741 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.605307 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:17.605458 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:17.605501 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:17.607704 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:60871: BlockingRecv error: recv got EOF from 127.1.1.195:60871 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:18.369628 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:18.365756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:52625)
0620 19:18:18.365915 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:18.365925 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:18.365970 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:18.366096 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:18.366097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:18.366241 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:18.366287 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.366288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.366902 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.366903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.367395 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.367396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.367676 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.367676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.367710 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.368099 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:18.368104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:18.369580 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
W20260620 19:18:18.369766 10245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:18.365831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52625 (local address 127.1.1.254:33367)
0620 19:18:18.365959 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:18.365961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:18.365969 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:18:18.366011 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:18.366012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:18.366021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:18.366079 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:18.366356 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.366840 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.367455 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.367621 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.367664 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.369663 (+ 1999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:52625: BlockingRecv error: recv got EOF from 127.1.1.193:52625 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:18.489132 10246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:18.484943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:37393)
0620 19:18:18.485040 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:18.485051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:18.485144 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:18.485253 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:18.485253 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:18.485399 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:18.485442 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.485443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.486068 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.486069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.486635 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.486637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.486943 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.486943 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.486981 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.487479 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:18.487487 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:18.489021 (+ 1534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":45,"threads_started":1}
W20260620 19:18:18.489166 10245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:18.485002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:37393 (local address 127.1.1.254:33367)
0620 19:18:18.485093 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:18.485095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:18.485135 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:18:18.485187 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:18.485188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:18.485196 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:18.485239 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:18.485492 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.485963 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.486701 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.486880 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.486931 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.489111 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:37393: BlockingRecv error: recv got EOF from 127.1.1.194:37393 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:18 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:18.612464 10247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:18.608153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:44165)
0620 19:18:18.608291 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:18.608302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:18.608394 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:18.608545 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:18.608546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:18.608692 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:18.608733 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.608735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.609370 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.609371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.609824 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:18.609826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.610253 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.610254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:18.610313 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.610772 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:18.610776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:18.612379 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260620 19:18:18.612512 10245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:18.608244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:44165 (local address 127.1.1.254:33367)
0620 19:18:18.608307 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:18.608309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:18.608378 (+ 69us) server_negotiation.cc:407] Connection header received
0620 19:18:18.608467 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:18.608468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:18.608479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:18.608538 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:18.608804 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.609300 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.609921 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:18.610173 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:18.610252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:18.612445 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:44165: BlockingRecv error: recv got EOF from 127.1.1.195:44165 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:19.378320 10248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:19.370046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:43663)
0620 19:18:19.370174 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:19.370187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:19.370249 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:19.370360 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:19.370361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:19.370498 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:19.370543 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.370544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.371092 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.371092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.371531 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.371532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.371797 (+ 265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.371798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.371831 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.376509 (+ 4678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:19.376521 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:19.378215 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
W20260620 19:18:19.378430 10249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:19.370083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:43663 (local address 127.1.1.254:33367)
0620 19:18:19.370211 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:19.370213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:19.370233 (+ 20us) server_negotiation.cc:407] Connection header received
0620 19:18:19.370284 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:19.370284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:19.370293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:19.370345 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:19.370593 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.371042 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.371589 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.371745 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.371787 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.378346 (+ 6559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:43663: BlockingRecv error: recv got EOF from 127.1.1.193:43663 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:19.493422 10250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:19.489537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:52175)
0620 19:18:19.489650 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:19.489659 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:19.489708 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:19.489822 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:19.489822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:19.489986 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:19.490033 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.490035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.490614 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.490615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.491112 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.491113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.491409 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.491410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.491442 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.491886 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:19.491891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:19.493369 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
W20260620 19:18:19.493523 10249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:19.489599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52175 (local address 127.1.1.254:33367)
0620 19:18:19.489660 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:19.489661 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:19.489704 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:18:19.489758 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:19.489758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:19.489768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:19.489806 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:19.490090 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.490554 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.491185 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.491354 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.491400 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.493456 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:52175: BlockingRecv error: recv got EOF from 127.1.1.194:52175 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:19 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:19.617508 10251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:19.612955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:40305)
0620 19:18:19.613178 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:19.613189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:19.613258 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:19.613383 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:19.613384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:19.613564 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:19.613607 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.613609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.614278 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.614279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.614827 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:19.614828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.615169 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.615170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:19.615225 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.615741 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:19.615746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:19.617442 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
W20260620 19:18:19.617570 10249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:19.613032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40305 (local address 127.1.1.254:33367)
0620 19:18:19.613159 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:19.613161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:19.613253 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:18:19.613311 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:19.613312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:19.613326 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:19.613378 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:19.613689 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.614194 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.614906 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:19.615081 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:19.615168 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:19.617526 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:40305: BlockingRecv error: recv got EOF from 127.1.1.195:40305 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:20.382776 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:20.378675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:38227)
0620 19:18:20.378864 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:20.378874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:20.378926 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:20.379044 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:20.379044 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:20.379191 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:20.379236 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.379238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.379843 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.379843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.380337 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.380338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.380615 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.380616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.380653 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.381051 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:20.381055 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:20.382695 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":124,"threads_started":1}
W20260620 19:18:20.382864 10253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:20.378798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:38227 (local address 127.1.1.254:33367)
0620 19:18:20.378937 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:20.378938 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:20.378945 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:18:20.378966 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:20.378966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:20.378975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:20.379027 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:20.379309 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.379780 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.380398 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.380559 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.380601 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.382798 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:38227: BlockingRecv error: recv got EOF from 127.1.1.193:38227 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:20.497979 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:20.493939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:52243)
0620 19:18:20.494068 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:20.494078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:20.494128 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:20.494250 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:20.494251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:20.494406 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:20.494466 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.494467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.495017 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.495018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.495480 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.495482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.495792 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.495793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.495829 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.496296 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:20.496301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:20.497927 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
W20260620 19:18:20.498059 10253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:20.494046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52243 (local address 127.1.1.254:33367)
0620 19:18:20.494113 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:20.494115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:20.494124 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:20.494177 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:20.494178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:20.494186 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:20.494234 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:20.494535 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.494959 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.495557 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.495730 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.495778 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.498002 (+ 2224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:52243: BlockingRecv error: recv got EOF from 127.1.1.194:52243 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:20 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:20.622241 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:20.617972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:46941)
0620 19:18:20.618160 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:20.618171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:20.618231 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:20.618420 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:20.618420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:20.618571 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:20.618617 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.618619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.619255 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.619256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.619815 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:20.619817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.620136 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.620136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:20.620172 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.620663 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:20.620667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:20.622180 (+ 1513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":113,"threads_started":1}
W20260620 19:18:20.622346 10253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:20.618046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:46941 (local address 127.1.1.254:33367)
0620 19:18:20.618103 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:20.618104 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:20.618309 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:18:20.618335 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:20.618336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:20.618348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:20.618405 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:20.618680 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.619172 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.619882 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:20.620060 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:20.620109 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:20.622257 (+ 2148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:46941: BlockingRecv error: recv got EOF from 127.1.1.195:46941 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:21.387480 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:21.383223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:52501)
0620 19:18:21.383367 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:21.383377 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:21.383424 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:21.383553 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:21.383554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:21.383688 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:21.383732 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.383733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.384326 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.384327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.384812 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.384814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.385156 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.385157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.385196 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.385631 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:21.385635 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:21.387430 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
W20260620 19:18:21.387593 10257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:21.383325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52501 (local address 127.1.1.254:33367)
0620 19:18:21.383455 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:21.383457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:21.383463 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:21.383480 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:21.383480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:21.383489 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:21.383537 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:21.383789 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.384260 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.384876 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.385041 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.385108 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.387514 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:52501: BlockingRecv error: recv got EOF from 127.1.1.193:52501 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:21.502470 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:21.498387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:40263)
0620 19:18:21.498496 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:21.498506 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:21.498566 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:21.498683 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:21.498683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:21.498895 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:21.498941 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.498943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.499545 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.499545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.500070 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.500071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.500380 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.500381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.500420 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.500807 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:21.500811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:21.502418 (+ 1607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
W20260620 19:18:21.502552 10257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:21.498446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:40263 (local address 127.1.1.254:33367)
0620 19:18:21.498487 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:21.498494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:21.498566 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:18:21.498620 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:21.498620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:21.498630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:21.498666 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:21.499004 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.499483 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.500130 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.500323 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.500364 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.502496 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:40263: BlockingRecv error: recv got EOF from 127.1.1.194:40263 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:21 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.195@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:21.627285 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:21.622601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.195:53769)
0620 19:18:21.622747 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:21.622756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:21.622822 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:21.622929 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:21.622930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:21.623069 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:21.623114 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.623115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.623666 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.623666 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.624455 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:21.624458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.624870 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.624872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:21.624927 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.625415 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:21.625419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:21.627219 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260620 19:18:21.627427 10257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:21.622671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:53769 (local address 127.1.1.254:33367)
0620 19:18:21.622726 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:21.622728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:21.622809 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:18:21.622859 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:21.622860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:21.622871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:21.622913 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:21.623165 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.623604 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.624549 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:21.624775 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:21.624864 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:21.627344 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.195:53769: BlockingRecv error: recv got EOF from 127.1.1.195:53769 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 19:18:22 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:22 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.193@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:22.392582 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.387962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.193:52919)
0620 19:18:22.388109 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:22.388123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:22.388191 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:22.388377 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:22.388377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:22.388554 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:22.388599 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.388600 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.389274 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.389275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.389872 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.389875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.390180 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.390180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.390231 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.390680 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:22.390685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:22.392513 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
W20260620 19:18:22.392649 10261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:22.388003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:52919 (local address 127.1.1.254:33367)
0620 19:18:22.388116 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:22.388118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:22.388175 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:18:22.388277 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:22.388278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:22.388290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:22.388358 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:22.388671 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.389205 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.389936 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.390105 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.390151 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.392596 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.193:52919: BlockingRecv error: recv got EOF from 127.1.1.193:52919 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
Jun 20 19:18:22 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:18:22 dist-test-slave-mwkb krb5kdc[9201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.1.194@KRBTEST.COM for oryx/127.1.1.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:18:22.507118 10262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.502941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:33367 (local address 127.1.1.194:44253)
0620 19:18:22.503083 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:22.503094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:22.503160 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:22.503282 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:22.503283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:22.503434 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:22.503480 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.503482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.504084 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.504085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.504593 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.504594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.504899 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.504900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.504938 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.505340 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:22.505344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:22.507027 (+ 1683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.1.254:33367: Server oryx/127.1.1.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260620 19:18:22.507165 10261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:22.503012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:44253 (local address 127.1.1.254:33367)
0620 19:18:22.503057 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:22.503059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:22.503153 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:18:22.503204 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:22.503205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:22.503216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:22.503262 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:22.503539 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.504024 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.504658 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.504842 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.504889 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.507102 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.1.194:44253: BlockingRecv error: recv got EOF from 127.1.1.194:44253 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:18:22.561017 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9785
I20260620 19:18:22.565546 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9936
I20260620 19:18:22.569458 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10070
I20260620 19:18:22.572996 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 9714
2026-06-20T19:18:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17219 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:22Z Disabled control of system clock
I20260620 19:18:22.598258 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34109
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:18:22.671821 10266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:22.671977 10266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:22.671994 10266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:22.672899 10266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:22.672927 10266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:22.672938 10266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:22.672950 10266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:22.672959 10266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:22.674012 10266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:34109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:22.674162 10266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:22.674336 10266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:22.678840 10272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.678849 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.678824 10271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:22.678818 10266 server_base.cc:1034] running on GCE node
I20260620 19:18:22.679276 10266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:22.679519 10266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:22.680650 10266 hybrid_clock.cc:648] HybridClock initialized: now 1781983102680643 us; error 28 us; skew 500 ppm
I20260620 19:18:22.681813 10266 webserver.cc:460] Webserver started at http://127.1.1.254:44669/ using document root <none> and password file <none>
I20260620 19:18:22.681967 10266 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:22.682001 10266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:22.682075 10266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:22.682907 10266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "f7f2963d20c345348a7f18fec4308034"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "cc2d12f56fb5f7864231ca42903d35f9"
server_key_iv: "6c3eeb2ab71e4140e75cff60c10c9b81"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.683146 10266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "f7f2963d20c345348a7f18fec4308034"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "cc2d12f56fb5f7864231ca42903d35f9"
server_key_iv: "6c3eeb2ab71e4140e75cff60c10c9b81"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.684149 10266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:22.684705 10280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.684932 10266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:22.685026 10266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "f7f2963d20c345348a7f18fec4308034"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "cc2d12f56fb5f7864231ca42903d35f9"
server_key_iv: "6c3eeb2ab71e4140e75cff60c10c9b81"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.685138 10266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:22.694087 10266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:22.694336 10266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:22.694468 10266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:22.698138 10266 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:34109
I20260620 19:18:22.698163 10332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:34109 every 8 connection(s)
I20260620 19:18:22.698447 10266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:22.699016 10333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:22.701210 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: Bootstrap starting.
I20260620 19:18:22.701697 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10266
I20260620 19:18:22.701719 10333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:22.701789 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:18:22.701962 1031 external_mini_cluster.cc:1428] Setting key e60738df459fddac681be068ba171fd3
I20260620 19:18:22.702008 10333 log.cc:824] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:22.702747 10333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: No bootstrap required, opened a new log
I20260620 19:18:22.704463 10333 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } }
I20260620 19:18:22.704604 10333 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:22.704643 10333 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f2963d20c345348a7f18fec4308034, State: Initialized, Role: FOLLOWER
I20260620 19:18:22.704761 10333 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } }
I20260620 19:18:22.705005 10333 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:22.705029 10333 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:22.705116 10333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:22.705664 10333 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } }
I20260620 19:18:22.705771 10333 leader_election.cc:302] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f2963d20c345348a7f18fec4308034; no voters:
I20260620 19:18:22.705893 10333 leader_election.cc:288] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:18:22.705955 10338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:22.706108 10333 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:22.706112 10338 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [term 1 LEADER]: Becoming Leader. State: Replica: f7f2963d20c345348a7f18fec4308034, State: Running, Role: LEADER
I20260620 19:18:22.706692 10338 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } }
I20260620 19:18:22.707319 10340 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f2963d20c345348a7f18fec4308034. Latest consensus state: current_term: 1 leader_uuid: "f7f2963d20c345348a7f18fec4308034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } } }
I20260620 19:18:22.707396 10340 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:22.707578 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.702385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.1.1.254:34109)
0620 19:18:22.702528 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:22.702535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:22.702562 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:18:22.703023 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:22.703030 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:22.703102 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:22.703268 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:22.703408 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.704582 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.705488 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.706027 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.706468 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.706815 (+ 347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:22.706818 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:22.706819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:22.707038 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:22.707076 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:22.707090 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:22.707151 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:22.707282 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:18:22.707361 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
I20260620 19:18:22.707593 10339 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f2963d20c345348a7f18fec4308034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f2963d20c345348a7f18fec4308034" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34109 } } }
I20260620 19:18:22.707893 10339 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034 [sys.catalog]: This master's current role is: LEADER
W20260620 19:18:22.707981 10353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:18:22.708047 10353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:18:22.708225 10352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:22.708396 10352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:18:22.709594 10352 catalog_manager.cc:1347] Generated new cluster ID: 31589d9092194fe190291a8a8908b129
I20260620 19:18:22.709640 10352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:18:22.716861 10352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:18:22.717599 10352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:22.720389 10352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7f2963d20c345348a7f18fec4308034: Generated new TSK 0
I20260620 19:18:22.720520 10352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:18:22.727974 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--builtin_ntp_servers=127.1.1.212:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:18:22.805986 10357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:22.806181 10357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:22.806211 10357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:22.807463 10357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:22.807524 10357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:22.807577 10357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:22.808936 10357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:22.809193 10357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:22.809477 10357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:22.814389 10362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.814479 10365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.814379 10363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:22.814675 10357 server_base.cc:1034] running on GCE node
I20260620 19:18:22.814904 10357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:22.815136 10357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:22.816309 10357 hybrid_clock.cc:648] HybridClock initialized: now 1781983102816292 us; error 31 us; skew 500 ppm
I20260620 19:18:22.817687 10357 webserver.cc:460] Webserver started at http://127.1.1.193:33133/ using document root <none> and password file <none>
I20260620 19:18:22.817915 10357 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:22.817982 10357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:22.818099 10357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:22.819011 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "d0f73fbdc62d4e059a277173a5d1bc91"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "7ece0d35d0246c708429e44514ccb4cb"
server_key_iv: "8945c78051fd34f38ee5463703417b0d"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.819355 10357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f73fbdc62d4e059a277173a5d1bc91"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "7ece0d35d0246c708429e44514ccb4cb"
server_key_iv: "8945c78051fd34f38ee5463703417b0d"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.820626 10357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:18:22.821411 10371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.821620 10357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:22.821710 10357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "d0f73fbdc62d4e059a277173a5d1bc91"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "7ece0d35d0246c708429e44514ccb4cb"
server_key_iv: "8945c78051fd34f38ee5463703417b0d"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.821774 10357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:22.831595 10357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:22.831827 10357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:22.831924 10357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:22.832079 10357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:22.832336 10357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:22.832365 10357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.832386 10357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:22.832397 10357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.837908 10357 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:45351
I20260620 19:18:22.837975 10484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:45351 every 8 connection(s)
I20260620 19:18:22.838274 10357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:18:22.843043 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.839043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:42309 (local address 127.1.1.254:34109)
0620 19:18:22.839102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:22.839104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:22.839326 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:18:22.839756 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:22.839757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:22.839769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:22.839829 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:22.840412 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.840877 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.841878 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.842064 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.842118 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.842805 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:22.842806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:22.842806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:22.842810 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:22.842821 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:22.842824 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:22.842866 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:22.842961 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:18:22.842991 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:18:22.843221 10487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.839040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34109 (local address 127.1.1.193:42309)
0620 19:18:22.839208 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:22.839238 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:22.839680 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:22.839877 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:22.839880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:22.839887 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:22.840343 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.840346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.840954 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.840954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.841807 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.841809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.842136 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.842137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.842234 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.842688 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:22.842689 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:22.842713 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:22.842732 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:22.842877 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:22.842880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:22.842883 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:22.842946 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:18:22.843024 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260620 19:18:22.843696 10485 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34109
I20260620 19:18:22.843780 10485 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:22.843935 10485 heartbeater.cc:511] Master 127.1.1.254:34109 requested a full tablet report, sending...
I20260620 19:18:22.844364 10297 ts_manager.cc:194] Registered new tserver with Master: d0f73fbdc62d4e059a277173a5d1bc91 (127.1.1.193:45351)
I20260620 19:18:22.844882 10297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.193:42309
I20260620 19:18:22.845590 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10357
I20260620 19:18:22.845652 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:18:22.845808 1031 external_mini_cluster.cc:1428] Setting key 54e4271ffa0e465aae03ce6f3ee69ee1
I20260620 19:18:22.846580 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--builtin_ntp_servers=127.1.1.212:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:18:22.918900 10488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:22.919068 10488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:22.919088 10488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:22.920137 10488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:22.920176 10488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:22.920197 10488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:22.921501 10488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:22.921720 10488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:22.921988 10488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:22.926558 10494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.926548 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:22.926548 10493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:22.926851 10488 server_base.cc:1034] running on GCE node
I20260620 19:18:22.927045 10488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:22.927233 10488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:22.928400 10488 hybrid_clock.cc:648] HybridClock initialized: now 1781983102928391 us; error 36 us; skew 500 ppm
I20260620 19:18:22.929688 10488 webserver.cc:460] Webserver started at http://127.1.1.194:33833/ using document root <none> and password file <none>
I20260620 19:18:22.929929 10488 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:22.929998 10488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:22.930203 10488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:22.931248 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "6b1e505bdacc4b66b1b8b2553eab398c"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "fd2bc12323a2ac174cc2eb12d8d24859"
server_key_iv: "9ec665d91adf2a6be22757ef2a39224e"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.931594 10488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "6b1e505bdacc4b66b1b8b2553eab398c"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "fd2bc12323a2ac174cc2eb12d8d24859"
server_key_iv: "9ec665d91adf2a6be22757ef2a39224e"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.932862 10488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:22.933609 10502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.933779 10488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.933871 10488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "6b1e505bdacc4b66b1b8b2553eab398c"
format_stamp: "Formatted at 2026-06-20 19:18:22 on dist-test-slave-mwkb"
server_key: "fd2bc12323a2ac174cc2eb12d8d24859"
server_key_iv: "9ec665d91adf2a6be22757ef2a39224e"
server_key_version: "encryptionkey@0"
I20260620 19:18:22.933954 10488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:22.941887 10488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:22.942133 10488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:22.942279 10488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:22.942483 10488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:22.942767 10488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:22.942816 10488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.942852 10488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:22.942876 10488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:22.948400 10488 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36259
I20260620 19:18:22.948436 10615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36259 every 8 connection(s)
I20260620 19:18:22.948745 10488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:18:22.950769 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10488
I20260620 19:18:22.950863 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:18:22.951058 1031 external_mini_cluster.cc:1428] Setting key d701eb090988863d66e8c138f2f86273
I20260620 19:18:22.952011 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--builtin_ntp_servers=127.1.1.212:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 19:18:22.954617 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.949461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:43597 (local address 127.1.1.254:34109)
0620 19:18:22.949508 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:22.949511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:22.949771 (+ 260us) server_negotiation.cc:407] Connection header received
0620 19:18:22.950191 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:22.950192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:22.950201 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:22.950251 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:22.950962 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.951634 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.953192 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.953377 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.953464 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.954333 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:22.954334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:22.954334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:22.954339 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:22.954360 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:22.954364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:22.954423 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:22.954516 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:18:22.954541 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:18:22.954813 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:22.949457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34109 (local address 127.1.1.194:43597)
0620 19:18:22.949639 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:22.949667 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:22.950117 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:22.950295 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:22.950298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:22.950305 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:22.950871 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.950878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.951733 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.951734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.952997 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:22.953000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:22.953485 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:22.953486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:22.953617 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:22.954181 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:22.954181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:22.954207 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:22.954227 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:22.954425 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:22.954431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:22.954434 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:18:22.954519 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:18:22.954614 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260620 19:18:22.955237 10616 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34109
I20260620 19:18:22.955328 10616 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:22.955508 10616 heartbeater.cc:511] Master 127.1.1.254:34109 requested a full tablet report, sending...
I20260620 19:18:22.955837 10297 ts_manager.cc:194] Registered new tserver with Master: 6b1e505bdacc4b66b1b8b2553eab398c (127.1.1.194:36259)
I20260620 19:18:22.956187 10297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.194:43597
W20260620 19:18:23.029644 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:23.029822 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:23.029842 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:23.030952 10619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:23.030992 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:23.031016 10619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:23.032212 10619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:23.032438 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:23.032744 10619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:23.037351 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.037361 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.037361 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.037628 10619 server_base.cc:1034] running on GCE node
I20260620 19:18:23.037880 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:23.038130 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:23.039286 10619 hybrid_clock.cc:648] HybridClock initialized: now 1781983103039287 us; error 40 us; skew 500 ppm
I20260620 19:18:23.040530 10619 webserver.cc:460] Webserver started at http://127.1.1.195:38153/ using document root <none> and password file <none>
I20260620 19:18:23.040782 10619 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:23.040849 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:23.041006 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:23.042014 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "3896fd3fcc134b4d9a75e8711aa01e05"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "67db8ca328e9dfb2b5ad261efc25e950"
server_key_iv: "047a0892b5a596076fa9927f500af823"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.042361 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "3896fd3fcc134b4d9a75e8711aa01e05"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "67db8ca328e9dfb2b5ad261efc25e950"
server_key_iv: "047a0892b5a596076fa9927f500af823"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.043589 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:23.044238 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.044385 10619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:23.044487 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "3896fd3fcc134b4d9a75e8711aa01e05"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "67db8ca328e9dfb2b5ad261efc25e950"
server_key_iv: "047a0892b5a596076fa9927f500af823"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.044566 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:23.052691 10619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:23.052947 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:23.053145 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:23.053349 10619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:23.053666 10619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:23.053720 10619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.053761 10619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:23.053787 10619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.059430 10619 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:44641
I20260620 19:18:23.059477 10746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:44641 every 8 connection(s)
I20260620 19:18:23.059819 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:18:23.064466 10337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.060473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:46683 (local address 127.1.1.254:34109)
0620 19:18:23.060514 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.060516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.060747 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:18:23.061276 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.061277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.061287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.061341 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.061975 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:23.062442 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:23.063333 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:23.063502 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:23.063547 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:23.064251 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:23.064252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:23.064252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:23.064256 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:23.064269 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:23.064272 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:23.064314 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:23.064392 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:18:23.064410 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:18:23.064630 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.060449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34109 (local address 127.1.1.195:46683)
0620 19:18:23.060627 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:23.060663 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:23.061195 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:23.061415 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:23.061418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:23.061426 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:23.061877 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:23.061880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:23.062516 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:23.062517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:23.063265 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:23.063266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:23.063567 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:23.063568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:23.063657 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:23.064130 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:23.064131 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:23.064155 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:23.064175 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:23.064325 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:23.064329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:23.064331 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:23.064382 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:18:23.064450 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260620 19:18:23.065112 10747 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34109
I20260620 19:18:23.065214 10747 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:23.065407 10747 heartbeater.cc:511] Master 127.1.1.254:34109 requested a full tablet report, sending...
I20260620 19:18:23.065757 10297 ts_manager.cc:194] Registered new tserver with Master: 3896fd3fcc134b4d9a75e8711aa01e05 (127.1.1.195:44641)
I20260620 19:18:23.066092 10297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.195:46683
I20260620 19:18:23.066908 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10619
I20260620 19:18:23.067006 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:18:23.067176 1031 external_mini_cluster.cc:1428] Setting key 4df1a68902c3f5989f870c34d60fc37a
I20260620 19:18:23.067987 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:18:23.069646 10337 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:23.069172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.1.1.254:34109)
0620 19:18:23.069229 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.069230 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.069374 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:18:23.069431 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.069432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.069440 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.069483 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.069598 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50790: BlockingRecv error: recv got EOF from 127.0.0.1:50790 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:18:23.069738 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10357
I20260620 19:18:23.074854 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10488
I20260620 19:18:23.078956 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10619
I20260620 19:18:23.082706 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10266
2026-06-20T19:18:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (508 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:23Z Disabled control of system clock
I20260620 19:18:23.107815 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37999
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:37999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:18:23.181975 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:23.182165 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:23.182194 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:23.183220 10758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:23.183296 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:23.183315 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:23.183336 10758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:23.183384 10758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:23.184656 10758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:37999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:23.184919 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:23.185259 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:23.189893 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.190065 10758 server_base.cc:1034] running on GCE node
W20260620 19:18:23.189901 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.189904 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.190454 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:23.190737 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:23.191900 10758 hybrid_clock.cc:648] HybridClock initialized: now 1781983103191876 us; error 57 us; skew 500 ppm
I20260620 19:18:23.193177 10758 webserver.cc:460] Webserver started at http://127.1.1.254:39967/ using document root <none> and password file <none>
I20260620 19:18:23.193436 10758 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:23.193503 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:23.193645 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:23.194579 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "b79b56856be84621b8dfc5c6674c7e2e"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "029384e1c1388e1b82bc6c81f67fa027"
server_key_iv: "c874c9eb398aaf4513315e10068e2560"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.194911 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "b79b56856be84621b8dfc5c6674c7e2e"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "029384e1c1388e1b82bc6c81f67fa027"
server_key_iv: "c874c9eb398aaf4513315e10068e2560"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.196074 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:23.196739 10772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.197000 10758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:23.197120 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "b79b56856be84621b8dfc5c6674c7e2e"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "029384e1c1388e1b82bc6c81f67fa027"
server_key_iv: "c874c9eb398aaf4513315e10068e2560"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.197194 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:23.200285 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:23.200510 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:23.200676 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:23.204084 10758 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:37999
I20260620 19:18:23.204123 10824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:37999 every 8 connection(s)
I20260620 19:18:23.204416 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:23.204960 10825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:23.207729 10825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e: Bootstrap starting.
I20260620 19:18:23.208274 10825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:23.208565 10825 log.cc:824] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:23.209223 10825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e: No bootstrap required, opened a new log
I20260620 19:18:23.210597 10825 raft_consensus.cc:348] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } }
I20260620 19:18:23.210711 10825 raft_consensus.cc:374] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:23.210776 10825 raft_consensus.cc:729] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79b56856be84621b8dfc5c6674c7e2e, State: Initialized, Role: FOLLOWER
I20260620 19:18:23.210920 10825 consensus_queue.cc:260] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } }
I20260620 19:18:23.210974 10825 raft_consensus.cc:388] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:23.210994 10825 raft_consensus.cc:482] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:23.211051 10825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:23.211318 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10758
I20260620 19:18:23.211388 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:18:23.211591 1031 external_mini_cluster.cc:1428] Setting key 28b9aecbeb12a431a89646abdc558a0d
I20260620 19:18:23.211649 10825 raft_consensus.cc:504] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } }
I20260620 19:18:23.211769 10825 leader_election.cc:302] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b79b56856be84621b8dfc5c6674c7e2e; no voters:
I20260620 19:18:23.211936 10825 leader_election.cc:288] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:18:23.211967 10828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:23.212121 10828 raft_consensus.cc:686] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [term 1 LEADER]: Becoming Leader. State: Replica: b79b56856be84621b8dfc5c6674c7e2e, State: Running, Role: LEADER
I20260620 19:18:23.212206 10825 sys_catalog.cc:564] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:23.212211 10828 consensus_queue.cc:237] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } }
I20260620 19:18:23.212872 10830 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b79b56856be84621b8dfc5c6674c7e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } } }
I20260620 19:18:23.213179 10830 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:23.213498 10842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:23.213060 10832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b79b56856be84621b8dfc5c6674c7e2e. Latest consensus state: current_term: 1 leader_uuid: "b79b56856be84621b8dfc5c6674c7e2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b79b56856be84621b8dfc5c6674c7e2e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37999 } } }
I20260620 19:18:23.214007 10832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:23.214123 10842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:18:23.214377 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.212018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.1.1.254:37999)
0620 19:18:23.212414 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.212423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.212601 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:18:23.212747 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.212754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.212821 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.213241 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.213867 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:23.213871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:23.213871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:23.213876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:23.213923 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:23.213938 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:23.214141 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:23.214174 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:18:23.214224 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":74,"threads_started":1}
I20260620 19:18:23.215535 10842 catalog_manager.cc:1347] Generated new cluster ID: d8fca4baf84b47da9321ee5a1b37603d
I20260620 19:18:23.215605 10842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:18:23.227689 10842 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:18:23.227855 10842 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:23.234542 10842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b79b56856be84621b8dfc5c6674c7e2e: Generated new TSK 0
I20260620 19:18:23.234730 10842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:18:23.243022 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--builtin_ntp_servers=127.1.1.212:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:18:23.337984 10849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:23.338182 10849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:23.338201 10849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:23.339316 10849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:23.339354 10849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:23.339380 10849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:23.340655 10849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:23.340864 10849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:23.341188 10849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:23.345906 10857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.345906 10854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.346097 10855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.346565 10849 server_base.cc:1034] running on GCE node
I20260620 19:18:23.346769 10849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:23.347008 10849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:23.348161 10849 hybrid_clock.cc:648] HybridClock initialized: now 1781983103348149 us; error 29 us; skew 500 ppm
I20260620 19:18:23.349323 10849 webserver.cc:460] Webserver started at http://127.1.1.193:43353/ using document root <none> and password file <none>
I20260620 19:18:23.349514 10849 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:23.349571 10849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:23.349681 10849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:23.350594 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "bbc5ac410f3c468e896433c3647baea0"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "d682b6ab3650d65885522df79691bad9"
server_key_iv: "9277dfe4d9a5f4e8c63bd775eba89580"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.350925 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "bbc5ac410f3c468e896433c3647baea0"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "d682b6ab3650d65885522df79691bad9"
server_key_iv: "9277dfe4d9a5f4e8c63bd775eba89580"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.352119 10849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:23.352792 10863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.352999 10849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:23.353104 10849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "bbc5ac410f3c468e896433c3647baea0"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "d682b6ab3650d65885522df79691bad9"
server_key_iv: "9277dfe4d9a5f4e8c63bd775eba89580"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.353191 10849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:23.357477 10849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:23.357738 10849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:23.357829 10849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:23.357966 10849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:23.358170 10849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:23.358196 10849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.358213 10849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:23.358224 10849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.363406 10849 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:43969
I20260620 19:18:23.363472 10976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:43969 every 8 connection(s)
I20260620 19:18:23.363718 10849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:18:23.366168 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.364319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:34643 (local address 127.1.1.254:37999)
0620 19:18:23.364380 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.364382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.364570 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:18:23.364976 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.364977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.364983 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.365031 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.365958 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:23.365960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:23.365960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:23.365964 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:23.365978 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:23.365982 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:23.366034 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:23.366097 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:18:23.366111 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:18:23.366344 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.364302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37999 (local address 127.1.1.193:34643)
0620 19:18:23.364473 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:23.364499 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:23.364884 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:23.365104 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:23.365108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:23.365117 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:23.365826 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:23.365827 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:23.365856 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:23.365877 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:23.366033 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:23.366038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:23.366040 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:23.366105 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:18:23.366153 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260620 19:18:23.366786 10977 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37999
I20260620 19:18:23.366858 10977 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:23.367015 10977 heartbeater.cc:511] Master 127.1.1.254:37999 requested a full tablet report, sending...
I20260620 19:18:23.367475 10789 ts_manager.cc:194] Registered new tserver with Master: bbc5ac410f3c468e896433c3647baea0 (127.1.1.193:43969)
I20260620 19:18:23.367880 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10849
I20260620 19:18:23.367950 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:18:23.368120 1031 external_mini_cluster.cc:1428] Setting key fca89c811c7afc72af7807ddbcbb90f3
I20260620 19:18:23.368846 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--builtin_ntp_servers=127.1.1.212:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:18:23.440461 10980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:23.440661 10980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:23.440696 10980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:23.441797 10980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:23.441864 10980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:23.441906 10980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:23.443200 10980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:23.443434 10980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:23.443677 10980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:18:23.448164 10980 server_base.cc:1034] running on GCE node
W20260620 19:18:23.448274 10986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.448166 10985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.448284 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.448554 10980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:23.448743 10980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:23.449883 10980 hybrid_clock.cc:648] HybridClock initialized: now 1781983103449868 us; error 31 us; skew 500 ppm
I20260620 19:18:23.450941 10980 webserver.cc:460] Webserver started at http://127.1.1.194:41641/ using document root <none> and password file <none>
I20260620 19:18:23.451195 10980 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:23.451251 10980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:23.451363 10980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:23.452165 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "f9be45c0c4964b8db40286d956e60654"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "80f0d28ccf90e732d035786e8b6605d2"
server_key_iv: "2ecc5ced405618ce7f051f505ff28a41"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.452435 10980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "f9be45c0c4964b8db40286d956e60654"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "80f0d28ccf90e732d035786e8b6605d2"
server_key_iv: "2ecc5ced405618ce7f051f505ff28a41"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.453565 10980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:23.454133 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.454276 10980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:23.454375 10980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "f9be45c0c4964b8db40286d956e60654"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "80f0d28ccf90e732d035786e8b6605d2"
server_key_iv: "2ecc5ced405618ce7f051f505ff28a41"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.454442 10980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:23.457567 10980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:23.457801 10980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:23.457927 10980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:23.458104 10980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:23.458360 10980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:23.458407 10980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.458439 10980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:23.458463 10980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.463711 10980 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:33129
I20260620 19:18:23.463760 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:33129 every 8 connection(s)
I20260620 19:18:23.464061 10980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:18:23.466687 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.464692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:60575 (local address 127.1.1.254:37999)
0620 19:18:23.464794 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.464797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.465025 (+ 228us) server_negotiation.cc:407] Connection header received
0620 19:18:23.465502 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.465503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.465508 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.465560 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.466517 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:23.466518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:23.466519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:23.466522 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:23.466535 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:23.466539 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:23.466582 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:23.466629 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:18:23.466640 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:18:23.466872 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.464666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37999 (local address 127.1.1.194:60575)
0620 19:18:23.464898 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:23.464926 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:23.465414 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:23.465604 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:23.465607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:23.465614 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:23.466399 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:23.466400 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:23.466423 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:23.466446 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:23.466579 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:23.466581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:23.466583 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:23.466630 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:18:23.466670 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":150,"threads_started":1}
I20260620 19:18:23.467322 11108 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37999
I20260620 19:18:23.467391 11108 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:23.467546 11108 heartbeater.cc:511] Master 127.1.1.254:37999 requested a full tablet report, sending...
I20260620 19:18:23.467883 10789 ts_manager.cc:194] Registered new tserver with Master: f9be45c0c4964b8db40286d956e60654 (127.1.1.194:33129)
I20260620 19:18:23.472867 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 10980
I20260620 19:18:23.472972 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:18:23.473153 1031 external_mini_cluster.cc:1428] Setting key aadaf8a6e5bacd18fa1f5244a14c2ff8
I20260620 19:18:23.473839 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--builtin_ntp_servers=127.1.1.212:36397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:18:23.547166 11111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:23.547349 11111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:23.547371 11111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:23.548347 11111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:23.548377 11111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:23.548417 11111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:23.549742 11111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:23.549938 11111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:23.550171 11111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:23.554616 11116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.554651 11117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:23.554647 11119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:23.554807 11111 server_base.cc:1034] running on GCE node
I20260620 19:18:23.555086 11111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:23.555292 11111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:23.556444 11111 hybrid_clock.cc:648] HybridClock initialized: now 1781983103556440 us; error 32 us; skew 500 ppm
I20260620 19:18:23.557818 11111 webserver.cc:460] Webserver started at http://127.1.1.195:46485/ using document root <none> and password file <none>
I20260620 19:18:23.558040 11111 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:23.558117 11111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:23.558320 11111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:23.559445 11111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "44686ffd3d554af89c8bf691df32161b"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "3bea7834888ec6df13f7e28258c5537a"
server_key_iv: "215152cbfebd9f35c889bb2e157a95ae"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.559779 11111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "44686ffd3d554af89c8bf691df32161b"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "3bea7834888ec6df13f7e28258c5537a"
server_key_iv: "215152cbfebd9f35c889bb2e157a95ae"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.561060 11111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:23.561803 11125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.561969 11111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:23.562050 11111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "44686ffd3d554af89c8bf691df32161b"
format_stamp: "Formatted at 2026-06-20 19:18:23 on dist-test-slave-mwkb"
server_key: "3bea7834888ec6df13f7e28258c5537a"
server_key_iv: "215152cbfebd9f35c889bb2e157a95ae"
server_key_version: "encryptionkey@0"
I20260620 19:18:23.562119 11111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:23.565302 11111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:23.565537 11111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:23.565665 11111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:23.565872 11111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:23.566165 11111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:23.566197 11111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.566251 11111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:23.566280 11111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:23.572010 11111 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:41953
I20260620 19:18:23.572072 11238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:41953 every 8 connection(s)
I20260620 19:18:23.572379 11111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:18:23.575032 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.573092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:53173 (local address 127.1.1.254:37999)
0620 19:18:23.573147 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.573148 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.573309 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:18:23.573702 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.573703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.573711 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.573790 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.574871 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:23.574871 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:23.574871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:18:23.574874 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:23.574884 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:18:23.574887 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:23.574927 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:23.574972 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:18:23.574982 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:18:23.575228 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:23.573033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37999 (local address 127.1.1.195:53173)
0620 19:18:23.573206 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:23.573241 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:23.573621 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:23.573845 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:23.573848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:23.573855 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:23.574743 (+ 888us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:18:23.574744 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:23.574768 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:18:23.574800 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:23.574926 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:23.574928 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:23.574930 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:23.574977 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:18:23.575018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260620 19:18:23.575693 11239 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37999
I20260620 19:18:23.575793 11239 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:23.575938 11239 heartbeater.cc:511] Master 127.1.1.254:37999 requested a full tablet report, sending...
I20260620 19:18:23.576254 10789 ts_manager.cc:194] Registered new tserver with Master: 44686ffd3d554af89c8bf691df32161b (127.1.1.195:41953)
I20260620 19:18:23.577567 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 11111
I20260620 19:18:23.577656 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:18:23.577842 1031 external_mini_cluster.cc:1428] Setting key 11c0521ea2a4ecf539ddc8a872ef7950
I20260620 19:18:23.578613 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:18:23.580273 10831 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:18:23.579837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.1.1.254:37999)
0620 19:18:23.579878 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:23.579880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:23.579965 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:18:23.580038 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:23.580039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:23.580044 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:23.580098 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:23.580216 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37432: BlockingRecv error: recv got EOF from 127.0.0.1:37432 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:18:23.580350 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10849
I20260620 19:18:23.584496 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10980
I20260620 19:18:23.588586 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 11111
I20260620 19:18:23.592268 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 10758
2026-06-20T19:18:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (509 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:18:23 dist-test-slave-mwkb krb5kdc[11249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:18:23 dist-test-slave-mwkb krb5kdc[11249](info): set up 2 sockets
Jun 20 19:18:23 dist-test-slave-mwkb krb5kdc[11249](info): commencing operation
krb5kdc: starting...
W20260620 19:18:24.588790 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:18:24 dist-test-slave-mwkb krb5kdc[11249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:18:24.697034 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:43521
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:43521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:24.771222 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:24.771420 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:24.771454 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:24.772470 11262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:18:24.772529 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:24.772553 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:24.772579 11262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:18:24.772605 11262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:18:24.773932 11262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:43521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:43521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:24.774159 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:24.774353 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:24.778568 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:24.778596 11262 server_base.cc:1034] running on GCE node
W20260620 19:18:24.778687 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:24.778962 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:24.779201 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:24.779426 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:24.780561 11262 hybrid_clock.cc:648] HybridClock initialized: now 1781983104780545 us; error 33 us; skew 500 ppm
Jun 20 19:18:24 dist-test-slave-mwkb krb5kdc[11249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:24.786470 11262 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:18:24.786895 11262 webserver.cc:460] Webserver started at http://127.1.1.254:40563/ using document root <none> and password file <none>
I20260620 19:18:24.787096 11262 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:24.787159 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:24.787299 11262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:24.788468 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "6cc990ac34be42b38dcda30bd0deb81d"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "574195099ca04d0723dea756b06e8d72"
server_key_iv: "a2b39bce468879f9d58a735b40fda78b"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.788892 11262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "6cc990ac34be42b38dcda30bd0deb81d"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "574195099ca04d0723dea756b06e8d72"
server_key_iv: "a2b39bce468879f9d58a735b40fda78b"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.790187 11262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:18:24.790889 11277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:24.791060 11262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:18:24.791141 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "6cc990ac34be42b38dcda30bd0deb81d"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "574195099ca04d0723dea756b06e8d72"
server_key_iv: "a2b39bce468879f9d58a735b40fda78b"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.791189 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:24.831907 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:24.834077 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:24.834184 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:24.837837 11262 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:43521
I20260620 19:18:24.837868 11329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:43521 every 8 connection(s)
I20260620 19:18:24.838202 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:18:24.839154 11330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:24.841264 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 11262
I20260620 19:18:24.841312 11330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: Bootstrap starting.
I20260620 19:18:24.841353 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:18:24.841522 1031 external_mini_cluster.cc:1428] Setting key 7d6bbf23b68a672d09f48d7c9a44a758
I20260620 19:18:24.841946 11330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:24.842232 11330 log.cc:824] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:24.842864 11330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: No bootstrap required, opened a new log
I20260620 19:18:24.844856 11330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } }
I20260620 19:18:24.845000 11330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:24.845027 11330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc990ac34be42b38dcda30bd0deb81d, State: Initialized, Role: FOLLOWER
I20260620 19:18:24.845168 11330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } }
I20260620 19:18:24.845264 11330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:18:24.845314 11330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:18:24.845379 11330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:24.845899 11330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } }
I20260620 19:18:24.846009 11330 leader_election.cc:302] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cc990ac34be42b38dcda30bd0deb81d; no voters:
I20260620 19:18:24.846201 11330 leader_election.cc:288] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:18:24.846222 11335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:24.846323 11335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [term 1 LEADER]: Becoming Leader. State: Replica: 6cc990ac34be42b38dcda30bd0deb81d, State: Running, Role: LEADER
I20260620 19:18:24.846417 11335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } }
I20260620 19:18:24.846454 11330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:18:24.846715 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cc990ac34be42b38dcda30bd0deb81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } } }
I20260620 19:18:24.846793 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:24.846920 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cc990ac34be42b38dcda30bd0deb81d. Latest consensus state: current_term: 1 leader_uuid: "6cc990ac34be42b38dcda30bd0deb81d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cc990ac34be42b38dcda30bd0deb81d" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 43521 } } }
I20260620 19:18:24.846989 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d [sys.catalog]: This master's current role is: LEADER
I20260620 19:18:24.847121 11340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:18:24.847755 11340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:18:24 dist-test-slave-mwkb krb5kdc[11249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:24.849036 11340 catalog_manager.cc:1347] Generated new cluster ID: b6a0fac84d6347edafa5ef0686258e74
I20260620 19:18:24.849458 11340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:18:24.851982 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:24.841975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57972 (local address 127.1.1.254:43521)
0620 19:18:24.842128 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:24.842139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:24.842160 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:18:24.842677 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:24.842691 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:24.842734 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:24.842837 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:24.843523 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:24.844257 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:24.845311 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:24.845756 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:24.845836 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:24.848439 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:24.848443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:24.848443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:24.848453 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:24.850227 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:24.850587 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:24.850590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:24.850590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:24.850670 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:24.850846 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:24.850847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:24.850847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:24.851090 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:24.851209 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:24.851626 (+ 417us) server_negotiation.cc:299] Negotiation successful
0620 19:18:24.851742 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260620 19:18:24.855185 11340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:18:24.855644 11340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:18:24.860360 11340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: Generated new TSK 0
I20260620 19:18:24.860502 11340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:18:24.885715 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:24.964268 11356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:24.964435 11356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:24.964455 11356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:24.965474 11356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:24.965508 11356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:24.965533 11356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:18:24.966812 11356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:24.966970 11356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:24.967123 11356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:24.971755 11362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:24.971707 11361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:24.971736 11364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:24.971938 11356 server_base.cc:1034] running on GCE node
I20260620 19:18:24.972121 11356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:24.972344 11356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:24.973495 11356 hybrid_clock.cc:648] HybridClock initialized: now 1781983104973480 us; error 24 us; skew 500 ppm
Jun 20 19:18:24 dist-test-slave-mwkb krb5kdc[11249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:24.975869 11356 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:18:24.976257 11356 webserver.cc:460] Webserver started at http://127.1.1.193:35295/ using document root <none> and password file <none>
I20260620 19:18:24.976475 11356 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:24.976539 11356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:24.976684 11356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:24.977702 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "6317e8ad4d944e6784fcc100bc779c2c"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "9f1b7ac0d687e9a0dc51db672ffdeb20"
server_key_iv: "417e0d3bf4939a0e5aacd3ad30fa9a30"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.978044 11356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "6317e8ad4d944e6784fcc100bc779c2c"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "9f1b7ac0d687e9a0dc51db672ffdeb20"
server_key_iv: "417e0d3bf4939a0e5aacd3ad30fa9a30"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.979354 11356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:18:24.980089 11371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:24.980299 11356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:24.980402 11356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "6317e8ad4d944e6784fcc100bc779c2c"
format_stamp: "Formatted at 2026-06-20 19:18:24 on dist-test-slave-mwkb"
server_key: "9f1b7ac0d687e9a0dc51db672ffdeb20"
server_key_iv: "417e0d3bf4939a0e5aacd3ad30fa9a30"
server_key_version: "encryptionkey@0"
I20260620 19:18:24.980480 11356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:25.004613 11356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:25.006268 11356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:25.006444 11356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:25.006654 11356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:25.006932 11356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:25.006983 11356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.007017 11356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:25.007045 11356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.012117 11356 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:38989
I20260620 19:18:25.012180 11484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:38989 every 8 connection(s)
I20260620 19:18:25.012468 11356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983104, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:25.019851 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 11356
I20260620 19:18:25.019956 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:18:25.020129 1031 external_mini_cluster.cc:1428] Setting key b53150eafcadc38af67bf14d05d7c10a
I20260620 19:18:25.020264 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.013091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:42219 (local address 127.1.1.254:43521)
0620 19:18:25.013136 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.013138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.013298 (+ 160us) server_negotiation.cc:407] Connection header received
0620 19:18:25.013670 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.013671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.013679 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.013724 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:25.014414 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.014862 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.015693 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.015870 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.015914 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.017913 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:25.017914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:25.017914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:25.017919 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:25.019051 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.019373 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.019374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.019374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.019418 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.019605 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.019606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.019606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.019710 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:25.019754 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.020071 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.020135 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:18:25.020474 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.013038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43521 (local address 127.1.1.193:42219)
0620 19:18:25.013190 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.013224 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.013603 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.013766 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.013768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.014209 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:25.014355 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.014358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.014936 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.014936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.015615 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.015616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.015931 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.015932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.016013 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.016506 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:25.016522 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:25.017832 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:25.019117 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.019122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.019124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.019313 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.019472 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.019472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.019473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.019555 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.019770 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:25.019782 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:25.019944 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.020046 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.020205 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260620 19:18:25.020853 11485 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43521
I20260620 19:18:25.020943 11485 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:25.021134 11485 heartbeater.cc:511] Master 127.1.1.254:43521 requested a full tablet report, sending...
I20260620 19:18:25.021607 11293 ts_manager.cc:194] Registered new tserver with Master: 6317e8ad4d944e6784fcc100bc779c2c (127.1.1.193:38989)
I20260620 19:18:25.022145 11293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:42219
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:18:25.041256 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:25.116385 11490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:25.116561 11490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:25.116583 11490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:25.117709 11490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:25.117763 11490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:25.117790 11490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:18:25.119150 11490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:25.119321 11490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:25.119474 11490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:25.124069 11496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:25.124061 11498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:25.124117 11490 server_base.cc:1034] running on GCE node
W20260620 19:18:25.124075 11495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:25.124473 11490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:25.124724 11490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:25.126006 11490 hybrid_clock.cc:648] HybridClock initialized: now 1781983105125984 us; error 34 us; skew 500 ppm
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:25.128721 11490 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:18:25.129242 11490 webserver.cc:460] Webserver started at http://127.1.1.194:43631/ using document root <none> and password file <none>
I20260620 19:18:25.129472 11490 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:25.129536 11490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:25.129671 11490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:25.130668 11490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "caa2b6b039ab49cc80a0b52b324beb7c"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "a0f45c29213365360f95d6265456f1b0"
server_key_iv: "283c60ba16af06177e3d1f5ab250cae9"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.131037 11490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "caa2b6b039ab49cc80a0b52b324beb7c"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "a0f45c29213365360f95d6265456f1b0"
server_key_iv: "283c60ba16af06177e3d1f5ab250cae9"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.132378 11490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:25.133195 11505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.133378 11490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:18:25.133478 11490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "caa2b6b039ab49cc80a0b52b324beb7c"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "a0f45c29213365360f95d6265456f1b0"
server_key_iv: "283c60ba16af06177e3d1f5ab250cae9"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.133563 11490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:25.143257 11490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:25.145218 11490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:25.145357 11490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:25.145548 11490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:25.145856 11490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:25.145910 11490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.145947 11490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:25.145972 11490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.151523 11490 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:40171
I20260620 19:18:25.151562 11618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:40171 every 8 connection(s)
I20260620 19:18:25.151806 11490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:18:25.154870 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 11490
I20260620 19:18:25.154947 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:18:25.155148 1031 external_mini_cluster.cc:1428] Setting key 8ade76030b194f1c25bffc0c7e7cdb9a
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:25.161171 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.152391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43521 (local address 127.1.1.194:33487)
0620 19:18:25.152537 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.152564 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.153041 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.153316 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.153320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.153868 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:25.154039 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.154044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.154755 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.154756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.155789 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.155791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.156164 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.156165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.156295 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.156764 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:25.156772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:25.158143 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:25.159909 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.159915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.159916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.160114 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.160330 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.160330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.160331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.160436 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.160723 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:25.160726 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:25.160824 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.160894 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.160991 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260620 19:18:25.161193 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.152397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:33487 (local address 127.1.1.254:43521)
0620 19:18:25.152438 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.152440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.152642 (+ 202us) server_negotiation.cc:407] Connection header received
0620 19:18:25.153187 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.153188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.153208 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.153258 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:25.154155 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.154644 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.155879 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.156074 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.156157 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.158220 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:25.158221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:25.158221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:25.158229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:25.159818 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.160187 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.160188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.160189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.160258 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.160514 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.160515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.160515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.160656 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:25.160735 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.160948 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.161034 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:18:25.161880 11619 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43521
I20260620 19:18:25.161985 11619 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:25.162138 11619 heartbeater.cc:511] Master 127.1.1.254:43521 requested a full tablet report, sending...
I20260620 19:18:25.162539 11293 ts_manager.cc:194] Registered new tserver with Master: caa2b6b039ab49cc80a0b52b324beb7c (127.1.1.194:40171)
I20260620 19:18:25.162948 11293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:33487
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:18:25.179229 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:44553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:18:25.254241 11624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:18:25.254412 11624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:18:25.254431 11624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:18:25.255436 11624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:18:25.255472 11624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:18:25.255515 11624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:18:25.257445 11624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:44553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:18:25.257679 11624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:18:25.257903 11624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:18:25.262149 11629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:25.262154 11632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:18:25.262246 11630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:18:25.262164 11624 server_base.cc:1034] running on GCE node
I20260620 19:18:25.262506 11624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:18:25.262742 11624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:18:25.263885 11624 hybrid_clock.cc:648] HybridClock initialized: now 1781983105263874 us; error 30 us; skew 500 ppm
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:18:25.266050 11624 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:18:25.266475 11624 webserver.cc:460] Webserver started at http://127.1.1.195:36855/ using document root <none> and password file <none>
I20260620 19:18:25.266691 11624 fs_manager.cc:359] Metadata directory not provided
I20260620 19:18:25.266777 11624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:18:25.266921 11624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:18:25.267848 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "1863b58a815d420889f4d602502ecedd"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "b5ee1e308019c5c190628162b4ec54f6"
server_key_iv: "80a96580a323f4376fdbfaa8637f1b5c"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.268146 11624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "1863b58a815d420889f4d602502ecedd"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "b5ee1e308019c5c190628162b4ec54f6"
server_key_iv: "80a96580a323f4376fdbfaa8637f1b5c"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.269349 11624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:18:25.270027 11639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.270180 11624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:18:25.270267 11624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "1863b58a815d420889f4d602502ecedd"
format_stamp: "Formatted at 2026-06-20 19:18:25 on dist-test-slave-mwkb"
server_key: "b5ee1e308019c5c190628162b4ec54f6"
server_key_iv: "80a96580a323f4376fdbfaa8637f1b5c"
server_key_version: "encryptionkey@0"
I20260620 19:18:25.270337 11624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:18:25.287700 11624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:18:25.289139 11624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:18:25.289266 11624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:18:25.289479 11624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:18:25.289754 11624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:18:25.289808 11624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.289846 11624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:18:25.289873 11624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:18:25.295509 11624 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:37691
I20260620 19:18:25.295552 11752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:37691 every 8 connection(s)
I20260620 19:18:25.295816 11624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:18:25.303113 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 11624
I20260620 19:18:25.303184 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:18:25.303387 1031 external_mini_cluster.cc:1428] Setting key 9fc4341aaa33efebba48ab489ec67edc
I20260620 19:18:25.303701 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.296388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:35837 (local address 127.1.1.254:43521)
0620 19:18:25.296449 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.296451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.296642 (+ 191us) server_negotiation.cc:407] Connection header received
0620 19:18:25.297026 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.297027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.297036 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.297148 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:25.297958 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.298407 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.299243 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.299401 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.299447 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.301248 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:25.301249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:25.301250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:25.301256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:25.302393 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.302824 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.302825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.302826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.302877 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.303085 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.303085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.303086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.303220 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:25.303272 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.303560 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.303625 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:18:25.303896 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.296369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:43521 (local address 127.1.1.195:35837)
0620 19:18:25.296536 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.296567 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.296952 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.297212 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.297216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.297726 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:18:25.297887 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.297891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.298485 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.298486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.299170 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.299172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.299463 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.299464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.299546 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.299977 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:18:25.299985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:18:25.301166 (+ 1181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:18:25.302484 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.302489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.302491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.302688 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.302934 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:18:25.302935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:18:25.302935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:18:25.303020 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:18:25.303298 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:18:25.303304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:18:25.303438 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.303537 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.303688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
I20260620 19:18:25.304268 11753 heartbeater.cc:347] Connected to a master server at 127.1.1.254:43521
I20260620 19:18:25.304348 11753 heartbeater.cc:464] Registering TS with master...
I20260620 19:18:25.304514 11753 heartbeater.cc:511] Master 127.1.1.254:43521 requested a full tablet report, sending...
I20260620 19:18:25.304829 11294 ts_manager.cc:194] Registered new tserver with Master: 1863b58a815d420889f4d602502ecedd (127.1.1.195:37691)
I20260620 19:18:25.305222 11294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:35837
I20260620 19:18:25.314996 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:18:25.321468 11333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.316179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.1.1.254:43521)
0620 19:18:25.316240 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.316242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.316307 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:18:25.316338 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.316338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.316345 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.316394 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:18:25.316899 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.317397 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.318073 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.318230 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.318274 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.319231 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:18:25.319232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:18:25.319233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:18:25.319238 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:18:25.320542 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.320888 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.320889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.320889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.320930 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:18:25.321049 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:18:25.321050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:18:25.321050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:18:25.321195 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:18:25.321266 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.321349 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.321396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 19:18:25.322680 11294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:18:25.324012 11294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:18:25.330417 11762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:37691 (local address 127.0.0.1:38956)
0620 19:18:25.326745 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.326756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.326811 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.327254 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.327255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.327257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:25.327309 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.327311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328153 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.328154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.329364 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.329366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.330102 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.330178 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330189 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.330279 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.330325 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":47,"threads_started":1}
I20260620 19:18:25.330420 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.1.1.195:37691)
0620 19:18:25.327096 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.327101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.327110 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:25.327141 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.327143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.327158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.327379 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:25.327439 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328040 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.329439 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.329996 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330098 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330151 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.330290 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.330325 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260620 19:18:25.330729 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:40171 (local address 127.0.0.1:59336)
0620 19:18:25.326877 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.326884 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.326919 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.327277 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.327278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.327280 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:25.327323 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.327324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328590 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.328591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.329588 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.329591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.330506 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.330570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330574 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.330603 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.330657 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":73,"threads_started":1}
I20260620 19:18:25.330823 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:38989 (local address 127.0.0.1:52214)
0620 19:18:25.326644 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.326679 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.326748 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.327182 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.327184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.327187 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:25.327266 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.327268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328554 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.328555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.329621 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.329623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.330564 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.330632 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330634 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.330686 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.330715 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260620 19:18:25.330837 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.1.1.193:38989)
0620 19:18:25.326908 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.326912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.326923 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:18:25.326964 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.326966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.326983 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.327091 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:25.327347 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328451 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.329725 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.330426 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330485 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330536 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.330717 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.330761 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260620 19:18:25.331100 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.326594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.1.1.194:40171)
0620 19:18:25.327110 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.327114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.327123 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:25.327160 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.327162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.327176 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.327394 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:25.327461 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.328512 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.329672 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.330398 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.330899 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.330963 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.330989 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.331018 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":68,"threads_started":1}
I20260620 19:18:25.331431 11419 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa952d68514245a4bf28b653601845 (DEFAULT_TABLE table=test-table [id=35eeae2167414d50a9fd7fcedb8c1bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:25.331630 11687 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa952d68514245a4bf28b653601845 (DEFAULT_TABLE table=test-table [id=35eeae2167414d50a9fd7fcedb8c1bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:25.331732 11419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa952d68514245a4bf28b653601845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:25.331915 11687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa952d68514245a4bf28b653601845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:25.334178 11767 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: Bootstrap starting.
I20260620 19:18:25.334488 11768 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Bootstrap starting.
I20260620 19:18:25.334681 11553 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfa952d68514245a4bf28b653601845 (DEFAULT_TABLE table=test-table [id=35eeae2167414d50a9fd7fcedb8c1bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:18:25.334823 11767 tablet_bootstrap.cc:654] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:25.335005 11553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfa952d68514245a4bf28b653601845. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:18:25.335207 11768 tablet_bootstrap.cc:654] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:25.335386 11767 log.cc:824] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:25.335731 11768 log.cc:824] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:25.336098 11767 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: No bootstrap required, opened a new log
I20260620 19:18:25.336184 11767 ts_tablet_manager.cc:1397] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:18:25.336400 11768 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: No bootstrap required, opened a new log
I20260620 19:18:25.336501 11768 ts_tablet_manager.cc:1397] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:18:25.337842 11771 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Bootstrap starting.
I20260620 19:18:25.338400 11771 tablet_bootstrap.cc:654] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Neither blocks nor log segments found. Creating new log.
I20260620 19:18:25.338295 11767 raft_consensus.cc:348] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.338459 11767 raft_consensus.cc:374] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:25.338387 11768 raft_consensus.cc:348] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.338502 11768 raft_consensus.cc:374] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:25.338501 11767 raft_consensus.cc:729] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6317e8ad4d944e6784fcc100bc779c2c, State: Initialized, Role: FOLLOWER
I20260620 19:18:25.338531 11768 raft_consensus.cc:729] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1863b58a815d420889f4d602502ecedd, State: Initialized, Role: FOLLOWER
I20260620 19:18:25.338620 11768 consensus_queue.cc:260] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.338637 11767 consensus_queue.cc:260] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.338927 11768 ts_tablet_manager.cc:1428] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:18:25.338927 11767 ts_tablet_manager.cc:1428] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:18:25.339004 11771 log.cc:824] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:18:25.339044 11485 heartbeater.cc:503] Master 127.1.1.254:43521 was elected leader, sending a full tablet report...
I20260620 19:18:25.339048 11753 heartbeater.cc:503] Master 127.1.1.254:43521 was elected leader, sending a full tablet report...
I20260620 19:18:25.339908 11771 tablet_bootstrap.cc:492] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: No bootstrap required, opened a new log
I20260620 19:18:25.339973 11771 ts_tablet_manager.cc:1397] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:18:25.341492 11771 raft_consensus.cc:348] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.341626 11771 raft_consensus.cc:374] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:18:25.341667 11771 raft_consensus.cc:729] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa2b6b039ab49cc80a0b52b324beb7c, State: Initialized, Role: FOLLOWER
I20260620 19:18:25.341766 11771 consensus_queue.cc:260] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.342001 11771 ts_tablet_manager.cc:1428] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:18:25.342031 11619 heartbeater.cc:503] Master 127.1.1.254:43521 was elected leader, sending a full tablet report...
I20260620 19:18:25.343871 11773 raft_consensus.cc:482] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:18:25.343940 11773 raft_consensus.cc:504] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.344252 11773 leader_election.cc:288] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa2b6b039ab49cc80a0b52b324beb7c (127.1.1.194:40171), 1863b58a815d420889f4d602502ecedd (127.1.1.195:37691)
I20260620 19:18:25.351065 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.344535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:44363 (local address 127.1.1.195:37691)
0620 19:18:25.344568 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.344570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.348623 (+ 4053us) server_negotiation.cc:407] Connection header received
0620 19:18:25.348698 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.348699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.348707 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.348746 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:25.348878 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.349342 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.350258 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.350737 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.350804 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.350811 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.350987 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.351015 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:18:25.351110 11776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.344374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:37691 (local address 127.1.1.193:44363)
0620 19:18:25.348548 (+ 4174us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.348560 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.348637 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.348753 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.348754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.348755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:25.348818 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.348820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.349435 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.349435 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.350169 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.350171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.350831 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.350832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.350914 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.350919 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.350969 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.351011 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4154,"thread_start_us":68,"threads_started":1}
I20260620 19:18:25.351466 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa952d68514245a4bf28b653601845" candidate_uuid: "6317e8ad4d944e6784fcc100bc779c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1863b58a815d420889f4d602502ecedd" is_pre_election: true
I20260620 19:18:25.351593 11707 raft_consensus.cc:2454] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6317e8ad4d944e6784fcc100bc779c2c in term 0.
I20260620 19:18:25.351828 11374 leader_election.cc:302] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1863b58a815d420889f4d602502ecedd, 6317e8ad4d944e6784fcc100bc779c2c; no voters:
I20260620 19:18:25.351941 11773 raft_consensus.cc:2781] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:18:25.351981 11773 raft_consensus.cc:482] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:18:25.352000 11773 raft_consensus.cc:3037] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:25.352416 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.344352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:40171 (local address 127.1.1.193:48839)
0620 19:18:25.344423 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.344431 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:18:25.344464 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:18:25.349390 (+ 4926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:18:25.349391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:18:25.349393 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:18:25.349449 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.349450 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.350336 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.350337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.351228 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:18:25.351230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.352120 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.352121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:18:25.352203 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.352207 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:18:25.352264 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:18:25.352313 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 19:18:25.352612 11773 raft_consensus.cc:504] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.352675 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.346136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:48839 (local address 127.1.1.194:40171)
0620 19:18:25.349299 (+ 3163us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.349300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.349306 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:18:25.349319 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.349320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.349327 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.349366 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:18:25.349521 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.350276 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.351319 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.352001 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.352077 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.352085 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.352555 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.352613 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3148}
I20260620 19:18:25.352730 11773 leader_election.cc:288] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [CANDIDATE]: Term 1 election: Requested vote from peers caa2b6b039ab49cc80a0b52b324beb7c (127.1.1.194:40171), 1863b58a815d420889f4d602502ecedd (127.1.1.195:37691)
I20260620 19:18:25.352912 11707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa952d68514245a4bf28b653601845" candidate_uuid: "6317e8ad4d944e6784fcc100bc779c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1863b58a815d420889f4d602502ecedd"
I20260620 19:18:25.352926 11573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa952d68514245a4bf28b653601845" candidate_uuid: "6317e8ad4d944e6784fcc100bc779c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" is_pre_election: true
I20260620 19:18:25.352938 11572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfa952d68514245a4bf28b653601845" candidate_uuid: "6317e8ad4d944e6784fcc100bc779c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa2b6b039ab49cc80a0b52b324beb7c"
I20260620 19:18:25.352978 11707 raft_consensus.cc:3037] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:25.353029 11573 raft_consensus.cc:2454] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6317e8ad4d944e6784fcc100bc779c2c in term 0.
I20260620 19:18:25.353649 11707 raft_consensus.cc:2454] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6317e8ad4d944e6784fcc100bc779c2c in term 1.
I20260620 19:18:25.353807 11374 leader_election.cc:302] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1863b58a815d420889f4d602502ecedd, 6317e8ad4d944e6784fcc100bc779c2c; no voters: caa2b6b039ab49cc80a0b52b324beb7c
I20260620 19:18:25.353900 11773 raft_consensus.cc:2781] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:18:25.354032 11773 raft_consensus.cc:686] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [term 1 LEADER]: Becoming Leader. State: Replica: 6317e8ad4d944e6784fcc100bc779c2c, State: Running, Role: LEADER
I20260620 19:18:25.354110 11773 consensus_queue.cc:237] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } }
I20260620 19:18:25.354746 11293 catalog_manager.cc:5495] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6317e8ad4d944e6784fcc100bc779c2c (127.1.1.193). New cstate: current_term: 1 leader_uuid: "6317e8ad4d944e6784fcc100bc779c2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6317e8ad4d944e6784fcc100bc779c2c" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 38989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 } health_report { overall_health: UNKNOWN } } }
I20260620 19:18:25.361619 11765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.359111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.1.1.193:38989)
0620 19:18:25.359143 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:18:25.359144 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:18:25.359153 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:18:25.359199 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:18:25.359200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:18:25.359216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:18:25.359297 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:18:25.359374 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.360134 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.360780 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:18:25.361093 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:18:25.361150 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:18:25.361351 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:18:25.361418 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:18:25.361490 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:18:25.361557 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:18:25.361571 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:18:25.363163 11572 raft_consensus.cc:3037] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:18:25.363240 11707 raft_consensus.cc:1261] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 1 FOLLOWER]: Refusing update from remote peer 6317e8ad4d944e6784fcc100bc779c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:25.363480 11773 consensus_queue.cc:1035] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1863b58a815d420889f4d602502ecedd" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:25.364091 11572 raft_consensus.cc:1261] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 1 FOLLOWER]: Refusing update from remote peer 6317e8ad4d944e6784fcc100bc779c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:18:25.364346 11773 consensus_queue.cc:1035] T 4bfa952d68514245a4bf28b653601845 P 6317e8ad4d944e6784fcc100bc779c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:18:25.365341 11780 mvcc.cc:205] Tried to move back new op lower bound from 7299002799565430784 to 7299002799530967040. Current Snapshot: MvccSnapshot[applied={T|T < 7299002799565430784}]
I20260620 19:18:25.365998 11781 mvcc.cc:205] Tried to move back new op lower bound from 7299002799565430784 to 7299002799530967040. Current Snapshot: MvccSnapshot[applied={T|T < 7299002799565430784}]
I20260620 19:18:25.368074 11294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:25.368155 11294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:18:25.368741 11294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cc990ac34be42b38dcda30bd0deb81d: Sending DeleteTablet for 3 replicas of tablet 4bfa952d68514245a4bf28b653601845
I20260620 19:18:25.369047 11419 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa952d68514245a4bf28b653601845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:25 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:52214
I20260620 19:18:25.369051 11687 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa952d68514245a4bf28b653601845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:25 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:38956
I20260620 19:18:25.369376 11553 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfa952d68514245a4bf28b653601845 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:18:25 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:59336
I20260620 19:18:25.369566 11790 tablet_replica.cc:330] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: stopping tablet replica
I20260620 19:18:25.369663 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 11356
I20260620 19:18:25.369695 11790 raft_consensus.cc:2229] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:25.371332 11790 pending_rounds.cc:62] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Trying to abort 2 pending ops.
I20260620 19:18:25.371424 11790 pending_rounds.cc:68] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7299002799565430784 op_type: WRITE_OP write_request { tablet_id: "4bfa952d68514245a4bf28b653601845" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\255\321\333\321\006\"8\n\ntest-admin\022*\n 35eeae2167414d50a9fd7fcedb8c1bfa\020\001\030\001 \001(\001" signature: "<redacted>""K\002V\327\371\003a\376zDO\223+\357\332\'\275cn\270i\371Y\364*)\206\"2\206\275\363l*\201\231Iq\353\023V\335\262n\205O\303\354L\034(\353\316\006\256\274C`JQ\373MCX" signing_key_seq_num: 0 } } request_id { client_id: "26fce4c6af314397987ab52cc53641e9" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 19:18:25.371644 11790 pending_rounds.cc:68] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7299002799530967040 op_type: NO_OP noop_request { }
I20260620 19:18:25.371738 11790 raft_consensus.cc:2866] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 19:18:25.371802 11790 raft_consensus.cc:2258] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:18:25.373629 11785 consensus_queue.cc:1159] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "caa2b6b039ab49cc80a0b52b324beb7c" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260620 19:18:25.373956 11788 tablet_replica.cc:330] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: stopping tablet replica
I20260620 19:18:25.374099 11788 raft_consensus.cc:2229] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:18:25.374226 11788 raft_consensus.cc:2258] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:18:25.374585 11509 connection.cc:535] server connection from 127.1.1.193:48839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:18:25.374668 11790 ts_tablet_manager.cc:1905] T 4bfa952d68514245a4bf28b653601845 P caa2b6b039ab49cc80a0b52b324beb7c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:18:25.374877 11281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:38989 (error 108)
I20260620 19:18:25.374894 11788 ts_tablet_manager.cc:1905] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:18:25.375505 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.375415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:38989 (local address 127.0.0.1:52224)
0620 19:18:25.375450 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.375477 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:38989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:18:25.375615 11281 catalog_manager.cc:4584] TS 6317e8ad4d944e6784fcc100bc779c2c (127.1.1.193:38989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bfa952d68514245a4bf28b653601845: Network error: Client connection negotiation failed: client connection to 127.1.1.193:38989: connect: Connection refused (error 111)
I20260620 19:18:25.375792 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 11490
I20260620 19:18:25.377059 11788 ts_tablet_manager.cc:1918] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:18:25.377149 11788 log.cc:1196] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/4bfa952d68514245a4bf28b653601845
I20260620 19:18:25.377430 11788 ts_tablet_manager.cc:1939] T 4bfa952d68514245a4bf28b653601845 P 1863b58a815d420889f4d602502ecedd: Deleting consensus metadata
I20260620 19:18:25.377876 11280 catalog_manager.cc:4841] TS 1863b58a815d420889f4d602502ecedd (127.1.1.195:37691): tablet 4bfa952d68514245a4bf28b653601845 (table test-table [id=35eeae2167414d50a9fd7fcedb8c1bfa]) successfully deleted
I20260620 19:18:25.381717 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:18:25.381622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:40171 (local address 127.0.0.1:59344)
0620 19:18:25.381665 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:18:25.381692 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.194:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:18:25.381745 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 11624
I20260620 19:18:25.385572 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 11262
2026-06-20T19:18:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1794 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11794](info): set up 2 sockets
Jun 20 19:18:25 dist-test-slave-mwkb krb5kdc[11794](info): commencing operation
krb5kdc: starting...
W20260620 19:18:26.384852 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:18:26 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:18:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangeradmin_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangeradmin_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangerlookup_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangerlookup_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
I20260620 19:18:26.523491 1031 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNtb4aU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/postgres -l logfile start
2026-06-20 19:18:28.629 UTC [11824] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:18:28.630 UTC [11824] LOG: listening on IPv4 address "127.1.1.212", port 46967
2026-06-20 19:18:28.635 UTC [11824] LOG: listening on Unix socket "/tmp/.s.PGSQL.46967"
2026-06-20 19:18:28.655 UTC [11827] LOG: database system was shut down at 2026-06-20 19:18:27 UTC
2026-06-20 19:18:28.658 UTC [11824] LOG: database system is ready to accept connections
I20260620 19:18:29.581007 1031 mini_postgres.cc:96] Postgres bound to 46967
2026-06-20 19:18:29.586 UTC [11835] FATAL: database "slave" does not exist
127.1.1.212:46967 - accepting connections
I20260620 19:18:29.587265 1031 mini_ranger.cc:161] Starting Ranger...
I20260620 19:18:29.601151 1031 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:18:29.999370 1031 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:18:29.999451 1031 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin
2026-06-20 19:18:30,262 [I] DB FLAVOR :POSTGRES
2026-06-20 19:18:30,263 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:18:30,263 [I] Checking connection..
2026-06-20 19:18:30,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:18:30,482 [I] Checking connection passed.
2026-06-20 19:18:30,482 [I] --------- Verifying version history table ---------
2026-06-20 19:18:30,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:18:30,730 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:18:30,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:18:30,956 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:18:30,956 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:18:30,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:18:31.172 UTC [11899] WARNING: there is no transaction in progress
2026-06-20 19:18:31,178 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:18:31,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:18:31,395 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:18:31,395 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:18:31,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:18:31,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:18:31,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mwkb.c.gcp-upstream.internal','N') ;"
2026-06-20 19:18:32,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:18:32,282 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:18:32,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:18:32,491 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:18:32,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:18:32,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:18:32,954 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:18:32,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:18:33.747 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.761 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.775 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.787 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.800 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.816 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.871 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.877 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.889 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.897 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.905 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.915 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.923 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.929 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.934 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:33.939 UTC [12007] WARNING: there is no transaction in progress
2026-06-20 19:18:34,430 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:18:34,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:18:34,663 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:18:34,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:18:34,879 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:18:34,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:18:35,091 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:18:35,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:18:35,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:18:35,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mwkb.c.gcp-upstream.internal';"
2026-06-20 19:18:35,742 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:18:35,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:18:35,955 [I] DB_PATCHES have already been applied
I20260620 19:18:35.960407 1031 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNtb4aU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:18:35.960458 1031 mini_ranger.cc:193] Using host: 127.1.1.212
I20260620 19:18:35.963019 1031 mini_ranger.cc:227] Ranger admin URL: http://127.1.1.212:46247
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:18:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:18:37.009963 1031 mini_ranger.cc:160] Time spent starting Ranger: real 7.423s user 0.000s sys 0.007s
Jun 20, 2026 7:18:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:18:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 7:18:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:18:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:18:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:18:55.936240 1031 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangerkms_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/rangerkms_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.1.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.1.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:18:55 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:18:56.001914 1031 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:18:56.009524 1031 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:18:56.730571 1031 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:18:56.730647 1031 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms
I20260620 19:18:56.730664 1031 mini_ranger_kms.cc:227] Using postgres at 127.1.1.212:46967
2026-06-20 19:18:56,889 [I] DB FLAVOR :POSTGRES
2026-06-20 19:18:56,890 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:18:56,890 [I] Checking connection
2026-06-20 19:18:56,890 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:18:57,111 [I] connection success
2026-06-20 19:18:57,111 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:18:57,111 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:18:57,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:18:57,322 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:18:57,322 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:18:57,322 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:18:57,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:46967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:18:57,544 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:18:57.971751 1031 mini_ranger_kms.cc:310] Created kms service
I20260620 19:18:58.143603 1031 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:18:58.180292 1031 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:18:58 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983135, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:18:58.541935 1031 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:18:58.542034 1031 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:18:58.542052 1031 mini_ranger_kms.cc:240] Using host: 127.1.1.212
I20260620 19:18:58.546186 1031 mini_ranger_kms.cc:276] Ranger KMS PID: 12187
I20260620 19:18:58.546406 1031 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.1.212:34371
Jun 20, 2026 7:18:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:18:59 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:18:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:18:59 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:19:00.659205 1031 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.657s user 0.005s sys 0.141s
I20260620 19:19:00.659310 1031 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:19:00.659384 1031 mini_ranger_kms.cc:401] 127.1.1.212:34371/kms/v1/keys
Jun 20 19:19:00 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781983140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:19:01 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
Jun 20 19:19:01 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983140, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
Jun 20 19:19:03 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:19:03.545799 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37123
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:37123
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:34371/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:03.621384 12238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:03.621562 12238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:03.621594 12238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:03.622592 12238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:03.622649 12238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:03.622673 12238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:03.622716 12238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:03.622747 12238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:03.624078 12238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.1.212:34371/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:37123
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:03.624332 12238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:03.624526 12238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:03.628938 12244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:03.629132 12238 server_base.cc:1034] running on GCE node
W20260620 19:19:03.628950 12243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:03.628954 12246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:03.629535 12238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:03.629822 12238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:03.630955 12238 hybrid_clock.cc:648] HybridClock initialized: now 1781983143630932 us; error 39 us; skew 500 ppm
Jun 20 19:19:03 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:03.633435 12238 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:03.633898 12238 webserver.cc:460] Webserver started at http://127.1.1.254:40065/ using document root <none> and password file <none>
I20260620 19:19:03.634100 12238 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:03.634148 12238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:03.634261 12238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:03 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983143, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:03.689724 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "7ad6db7ad3b14bde818902200c50dec2"
format_stamp: "Formatted at 2026-06-20 19:19:03 on dist-test-slave-mwkb"
server_key: "b64859799ed5d832c05f172e00d2ee41"
server_key_iv: "8a13e207111703ba7812150f91a957ee"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:03.690115 12238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "7ad6db7ad3b14bde818902200c50dec2"
format_stamp: "Formatted at 2026-06-20 19:19:03 on dist-test-slave-mwkb"
server_key: "b64859799ed5d832c05f172e00d2ee41"
server_key_iv: "8a13e207111703ba7812150f91a957ee"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:03.691720 12238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:19:03.704888 12253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:03.705092 12238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:19:03.705212 12238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "7ad6db7ad3b14bde818902200c50dec2"
format_stamp: "Formatted at 2026-06-20 19:19:03 on dist-test-slave-mwkb"
server_key: "b64859799ed5d832c05f172e00d2ee41"
server_key_iv: "8a13e207111703ba7812150f91a957ee"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:03.705387 12238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:03.714660 12238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:03.716706 12238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:03.716850 12238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:03.721621 12238 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:37123
I20260620 19:19:03.721638 12305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:37123 every 8 connection(s)
I20260620 19:19:03.722020 12238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:03.722595 12306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:03.724502 12306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: Bootstrap starting.
I20260620 19:19:03.725019 12306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:03.725358 12306 log.cc:824] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:03.726033 12306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: No bootstrap required, opened a new log
I20260620 19:19:03.728310 12306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } }
I20260620 19:19:03.728443 12306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:03.728483 12306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad6db7ad3b14bde818902200c50dec2, State: Initialized, Role: FOLLOWER
I20260620 19:19:03.728609 12306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } }
I20260620 19:19:03.728732 12306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:03.728771 12306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:03.728812 12306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:03.729306 12306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } }
I20260620 19:19:03.729408 12306 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad6db7ad3b14bde818902200c50dec2; no voters:
I20260620 19:19:03.729604 12306 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:19:03.729651 12309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:03.729836 12309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [term 1 LEADER]: Becoming Leader. State: Replica: 7ad6db7ad3b14bde818902200c50dec2, State: Running, Role: LEADER
I20260620 19:19:03.729857 12306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:03.729955 12309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } }
I20260620 19:19:03.730438 12309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad6db7ad3b14bde818902200c50dec2. Latest consensus state: current_term: 1 leader_uuid: "7ad6db7ad3b14bde818902200c50dec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } } }
I20260620 19:19:03.730564 12309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:03.730448 12311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad6db7ad3b14bde818902200c50dec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad6db7ad3b14bde818902200c50dec2" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 37123 } } }
I20260620 19:19:03.730651 12311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:03.731396 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 12238
I20260620 19:19:03.731518 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:19:03.731691 1031 external_mini_cluster.cc:1428] Setting key 9c627353b4fff218ea753d042af8c46b
I20260620 19:19:03.731935 12318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:03.732175 12306 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:19:03.732399 12318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:03.737567 12318 catalog_manager.cc:1347] Generated new cluster ID: 17fb0ac3fe064aeb8e7140c7b645f3e4
I20260620 19:19:03.737694 12318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:03.744123 12318 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:19:03.744774 12318 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 19:19:03 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:03.759160 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:03.734099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.1.1.254:37123)
0620 19:19:03.734716 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:03.734725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:03.743985 (+ 9260us) server_negotiation.cc:407] Connection header received
0620 19:19:03.744114 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:03.744120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:03.744168 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:03.744283 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:03.749876 (+ 5593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:03.751460 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:03.752481 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:03.753028 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:03.753176 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:03.755779 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:03.755782 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:03.755782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:03.755794 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:03.757603 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:03.758021 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:03.758024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:03.758024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:03.758115 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:03.758297 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:03.758298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:03.758298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:03.758588 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:03.758704 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:03.758835 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:19:03.758965 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":221,"threads_started":1}
I20260620 19:19:03.766139 12318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: Generated new TSK 0
I20260620 19:19:03.766323 12318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:19:04.392287 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:34371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:04.473698 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:04.473888 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:04.473922 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:04.474949 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:04.475013 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:04.475054 12351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:04.476392 12351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.1.212:34371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:04.476612 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:04.476799 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:04.482100 12356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:04.482110 12357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:04.482343 12351 server_base.cc:1034] running on GCE node
W20260620 19:19:04.482129 12359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:04.482591 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:04.482849 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:04.484019 12351 hybrid_clock.cc:648] HybridClock initialized: now 1781983144484007 us; error 26 us; skew 500 ppm
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:04.491711 12351 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:04.492122 12351 webserver.cc:460] Webserver started at http://127.1.1.193:34411/ using document root <none> and password file <none>
I20260620 19:19:04.492336 12351 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:04.492400 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:04.492539 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:04.508509 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "f4efd5ea8256434087cf85ef00e7d440"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "42c17ad491ef602483bbb31526e4d987"
server_key_iv: "13821dcfad0570ce218a8182efc08d8b"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.509013 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "f4efd5ea8256434087cf85ef00e7d440"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "42c17ad491ef602483bbb31526e4d987"
server_key_iv: "13821dcfad0570ce218a8182efc08d8b"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.510941 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:19:04.527983 12366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.528194 12351 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 19:19:04.528290 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "f4efd5ea8256434087cf85ef00e7d440"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "42c17ad491ef602483bbb31526e4d987"
server_key_iv: "13821dcfad0570ce218a8182efc08d8b"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.528383 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:04.573032 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:04.575281 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:04.575420 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:04.575640 12351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:04.575968 12351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:04.576011 12351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.576050 12351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:04.576099 12351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.583668 12351 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:34887
I20260620 19:19:04.583750 12481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:34887 every 8 connection(s)
I20260620 19:19:04.584025 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:19:04.588119 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 12351
I20260620 19:19:04.588217 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:19:04.588367 1031 external_mini_cluster.cc:1428] Setting key 68eb50febbc54a0ea991993f0ccef3ad
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:04.594729 12484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:04.584647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37123 (local address 127.1.1.193:60985)
0620 19:19:04.584801 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:04.584844 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:04.585399 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:04.585694 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:04.585698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:04.585907 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:04.586143 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:04.586150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.586882 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.586883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:04.588045 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:04.588048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.588559 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.588561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:04.588688 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:04.589156 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:04.589165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:04.591142 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:04.593194 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:04.593200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:04.593201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:04.593497 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:04.593714 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:04.593715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:04.593715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:04.593821 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:04.594095 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:04.594101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:04.594257 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 19:19:04.594355 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:19:04.594498 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
I20260620 19:19:04.595161 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:04.584904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60985 (local address 127.1.1.254:37123)
0620 19:19:04.585047 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:04.585049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:04.585059 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:19:04.585505 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:04.585506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:04.585520 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:04.585615 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:04.586260 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.586782 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.588128 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.588414 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.588497 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:04.591236 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:04.591238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:04.591238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:04.591246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:04.593114 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:04.593580 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:04.593581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:04.593582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:04.593641 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:04.593883 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:04.593884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:04.593884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:04.594010 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:04.594068 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:04.594960 (+ 892us) server_negotiation.cc:299] Negotiation successful
0620 19:19:04.595032 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260620 19:19:04.595515 12482 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37123
I20260620 19:19:04.595628 12482 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:04.595785 12482 heartbeater.cc:511] Master 127.1.1.254:37123 requested a full tablet report, sending...
I20260620 19:19:04.596271 12270 ts_manager.cc:194] Registered new tserver with Master: f4efd5ea8256434087cf85ef00e7d440 (127.1.1.193:34887)
I20260620 19:19:04.597167 12270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:60985
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:19:04.615250 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:34371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:04.722054 12488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:04.722349 12488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:04.722424 12488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:04.723953 12488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:04.724053 12488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:04.724131 12488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:04.726130 12488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.1.212:34371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:04.726327 12488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:04.726590 12488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:04.732501 12495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:04.732813 12498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:04.733012 12496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:04.733172 12488 server_base.cc:1034] running on GCE node
I20260620 19:19:04.733560 12488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:04.733774 12488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:04.734937 12488 hybrid_clock.cc:648] HybridClock initialized: now 1781983144734917 us; error 31 us; skew 500 ppm
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:04.737676 12488 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:04.738170 12488 webserver.cc:460] Webserver started at http://127.1.1.194:45095/ using document root <none> and password file <none>
I20260620 19:19:04.738442 12488 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:04.738504 12488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:04.738673 12488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:04.760211 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "cbb1f8b3b9a64613b1959ec66ae06962"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "6b277da8543a24414394e33654be51b0"
server_key_iv: "5a9e62c351b9950fabc37374fa4de160"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.760658 12488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "cbb1f8b3b9a64613b1959ec66ae06962"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "6b277da8543a24414394e33654be51b0"
server_key_iv: "5a9e62c351b9950fabc37374fa4de160"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.768539 12488 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.003s
I20260620 19:19:04.781570 12505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.785236 12488 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260620 19:19:04.785326 12488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "cbb1f8b3b9a64613b1959ec66ae06962"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "6b277da8543a24414394e33654be51b0"
server_key_iv: "5a9e62c351b9950fabc37374fa4de160"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.785416 12488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:04.804548 12488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:04.806552 12488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:04.806689 12488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:04.806912 12488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:04.807245 12488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:04.807291 12488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.807351 12488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:04.807379 12488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:04.814177 12488 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36015
I20260620 19:19:04.814257 12619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36015 every 8 connection(s)
I20260620 19:19:04.814554 12488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:19:04.821117 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 12488
I20260620 19:19:04.821206 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:19:04.821389 1031 external_mini_cluster.cc:1428] Setting key 410d57827e100e6b69bec91c7e947b9a
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:04.831954 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:04.815206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:42597 (local address 127.1.1.254:37123)
0620 19:19:04.815246 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:04.815247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:04.821885 (+ 6638us) server_negotiation.cc:407] Connection header received
0620 19:19:04.822533 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:04.822534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:04.822545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:04.822574 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:04.823317 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.824089 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.825482 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.825777 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.825815 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:04.828127 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:04.828128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:04.828128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:04.828134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:04.829709 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:04.830930 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:04.830931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:04.830931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:04.830987 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:04.831250 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:04.831251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:04.831251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:04.831390 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:04.831430 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:04.831777 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:19:04.831861 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:19:04.832614 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:04.815201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37123 (local address 127.1.1.194:42597)
0620 19:19:04.821764 (+ 6563us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:04.821807 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:04.822414 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:04.822738 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:04.822743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:04.822961 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:04.823200 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:04.823206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.824198 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.824199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:04.825387 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:04.825389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:04.825914 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:04.825914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:04.826047 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:04.826555 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:04.826564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:04.828030 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:04.829797 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:04.829803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:04.829805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:04.830837 (+ 1032us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:04.831057 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:04.831058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:04.831058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:04.831178 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:04.831474 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:04.831479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:04.831632 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 19:19:04.832260 (+ 628us) client_negotiation.cc:240] Negotiation successful
0620 19:19:04.832412 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6505,"thread_start_us":75,"threads_started":1}
I20260620 19:19:04.833223 12620 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37123
I20260620 19:19:04.833310 12620 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:04.833475 12620 heartbeater.cc:511] Master 127.1.1.254:37123 requested a full tablet report, sending...
I20260620 19:19:04.833793 12270 ts_manager.cc:194] Registered new tserver with Master: cbb1f8b3b9a64613b1959ec66ae06962 (127.1.1.194:36015)
I20260620 19:19:04.834256 12270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:42597
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:19:04.859737 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:34371/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:04.952580 12625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:04.952785 12625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:04.952823 12625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:04.954044 12625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:04.954113 12625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:04.954157 12625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:04.955564 12625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.1.212:34371/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:37123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:04.955797 12625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:04.956022 12625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:04.960790 12631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:04.960891 12630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:04.960790 12633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:04.961836 12625 server_base.cc:1034] running on GCE node
I20260620 19:19:04.962059 12625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:04.962311 12625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:04.965590 12625 hybrid_clock.cc:648] HybridClock initialized: now 1781983144965568 us; error 33 us; skew 500 ppm
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:04.968691 12625 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:04.969161 12625 webserver.cc:460] Webserver started at http://127.1.1.195:40715/ using document root <none> and password file <none>
I20260620 19:19:04.969400 12625 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:04.969452 12625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:04.969576 12625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:04 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:04.986498 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "61bf32123932d2c27a3c0e66cc4a1e5d"
server_key_iv: "a5610a1594b69e4e530c9245b526dde3"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.986907 12625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "61bf32123932d2c27a3c0e66cc4a1e5d"
server_key_iv: "a5610a1594b69e4e530c9245b526dde3"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:04.990110 12625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 19:19:05.007742 12640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:05.007957 12625 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:05.008044 12625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0"
format_stamp: "Formatted at 2026-06-20 19:19:04 on dist-test-slave-mwkb"
server_key: "61bf32123932d2c27a3c0e66cc4a1e5d"
server_key_iv: "a5610a1594b69e4e530c9245b526dde3"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:05.008121 12625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:05.050585 12625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:05.052982 12625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:05.053188 12625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:05.053457 12625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:05.053812 12625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:05.053905 12625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:05.053960 12625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:05.054001 12625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:05.061357 12625 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:32985
I20260620 19:19:05.061411 12753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:32985 every 8 connection(s)
I20260620 19:19:05.061842 12625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:19:05 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:05.072001 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 12625
I20260620 19:19:05.072352 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:19:05.072817 1031 external_mini_cluster.cc:1428] Setting key 4b9518381318f8e85016244ce6603477
I20260620 19:19:05.074915 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.062576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:50701 (local address 127.1.1.254:37123)
0620 19:19:05.062634 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.062636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.063109 (+ 473us) server_negotiation.cc:407] Connection header received
0620 19:19:05.063791 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.063792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.063804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.063847 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:05.064605 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.065351 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.066657 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.066937 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.066976 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.071759 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:05.071760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:05.071761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:05.071767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:05.073398 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:05.073847 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:05.073848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:05.073849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:05.073908 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:05.074185 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:05.074186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:05.074186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:05.074313 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:05.074368 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.074722 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.074799 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:19:05.075389 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.062558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:37123 (local address 127.1.1.195:50701)
0620 19:19:05.062950 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.062994 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.063655 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.064020 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.064024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.064237 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:05.064492 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.064499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.065454 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.065454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.066564 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.066567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.067059 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.067060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.069281 (+ 2221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.069788 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:05.069798 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:05.071635 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:05.073479 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:05.073485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:05.073487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:05.073746 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:05.073985 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:05.073986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:05.073986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:05.074106 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:05.074414 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:05.074419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:05.074573 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.075057 (+ 484us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.075192 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":84,"threads_started":1}
I20260620 19:19:05.075765 12754 heartbeater.cc:347] Connected to a master server at 127.1.1.254:37123
I20260620 19:19:05.075858 12754 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:05.076018 12754 heartbeater.cc:511] Master 127.1.1.254:37123 requested a full tablet report, sending...
I20260620 19:19:05.076320 12270 ts_manager.cc:194] Registered new tserver with Master: 9c7ff4e750b8456aac9cc010dfdf4cf0 (127.1.1.195:32985)
I20260620 19:19:05.076817 12270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:50701
I20260620 19:19:05.084452 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:19:05.094177 12485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.086023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.1.1.254:37123)
0620 19:19:05.086059 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.086060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.086239 (+ 179us) server_negotiation.cc:407] Connection header received
0620 19:19:05.086349 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.086350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.086359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.086387 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:05.088012 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.088650 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.089790 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.090074 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.090116 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.091540 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:05.091541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:05.091541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:05.091547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:05.092926 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:05.093325 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:05.093326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:05.093326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:05.093378 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:05.093567 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:05.093568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:05.093568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:05.093681 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:05.093750 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.094022 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.094093 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:19:05.096238 12270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:19:05.098394 12270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:19:05.106930 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.100930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:32985 (local address 127.0.0.1:42650)
0620 19:19:05.101959 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.101996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.102060 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.102324 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.102326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.102328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:05.102382 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.102384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.103880 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.103881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.105211 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.105214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.106639 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.106640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.106718 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.106730 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.106795 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.106854 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"thread_start_us":75,"threads_started":1}
I20260620 19:19:05.107306 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.101251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.1.1.195:32985)
0620 19:19:05.101902 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.101906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.102136 (+ 230us) server_negotiation.cc:407] Connection header received
0620 19:19:05.102173 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.102175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.102190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.102253 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:05.102458 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.103788 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.105364 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.106429 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.106523 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.106606 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.107193 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.107232 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":71,"threads_started":1}
I20260620 19:19:05.108177 12670 tablet_service.cc:1467] Processing CreateTablet for tablet fce042fa402441249a1643ea9823be92 (DEFAULT_TABLE table=test-table [id=582c2138d4374c8284594987829a7320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:05.108559 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce042fa402441249a1643ea9823be92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:05.110021 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.101156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34887 (local address 127.0.0.1:59708)
0620 19:19:05.105170 (+ 4014us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.105180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.105241 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.105501 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.105502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.105503 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:05.105541 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.105542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.107283 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.107284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.108472 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.108475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.109778 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.109779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.109855 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.109860 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.109915 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.109954 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3998,"thread_start_us":60,"threads_started":1}
I20260620 19:19:05.110272 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.101394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.1.1.193:34887)
0620 19:19:05.101821 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.101826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.105238 (+ 3412us) server_negotiation.cc:407] Connection header received
0620 19:19:05.105339 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.105342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.105359 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.105412 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:05.105618 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.107172 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.108589 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.109601 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.109690 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.109742 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.110164 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.110206 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":69,"threads_started":1}
I20260620 19:19:05.110865 12416 tablet_service.cc:1467] Processing CreateTablet for tablet fce042fa402441249a1643ea9823be92 (DEFAULT_TABLE table=test-table [id=582c2138d4374c8284594987829a7320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:05.111171 12416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce042fa402441249a1643ea9823be92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:05.114756 12773 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Bootstrap starting.
I20260620 19:19:05.115465 12773 tablet_bootstrap.cc:654] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:05.115801 12773 log.cc:824] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:05.116155 12772 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Bootstrap starting.
I20260620 19:19:05.116969 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.100956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36015 (local address 127.0.0.1:37934)
0620 19:19:05.104815 (+ 3859us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.104827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.104878 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.113366 (+ 8488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.113367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.113369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:05.113405 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.113407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.114646 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.114647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.115676 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.115679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.116760 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.116761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.116814 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.116817 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.116872 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.116908 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3841,"thread_start_us":69,"threads_started":1}
I20260620 19:19:05.117158 12772 tablet_bootstrap.cc:654] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:05.117460 12772 log.cc:824] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:05.117550 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.101344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.1.1.194:36015)
0620 19:19:05.113202 (+ 11858us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.113207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.113217 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:19:05.113253 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.113254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.113268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.113465 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:05.113529 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.114544 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.116032 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.116668 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.117326 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.117408 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.117435 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.117469 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11820,"thread_start_us":81,"threads_started":1}
I20260620 19:19:05.118240 12772 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: No bootstrap required, opened a new log
I20260620 19:19:05.118348 12772 ts_tablet_manager.cc:1397] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:05.118258 12548 tablet_service.cc:1467] Processing CreateTablet for tablet fce042fa402441249a1643ea9823be92 (DEFAULT_TABLE table=test-table [id=582c2138d4374c8284594987829a7320]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:05.118613 12548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce042fa402441249a1643ea9823be92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:05.120266 12772 raft_consensus.cc:348] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.120436 12772 raft_consensus.cc:374] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:05.120471 12772 raft_consensus.cc:729] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c7ff4e750b8456aac9cc010dfdf4cf0, State: Initialized, Role: FOLLOWER
I20260620 19:19:05.120571 12772 consensus_queue.cc:260] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.120815 12772 ts_tablet_manager.cc:1428] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:19:05.120942 12754 heartbeater.cc:503] Master 127.1.1.254:37123 was elected leader, sending a full tablet report...
I20260620 19:19:05.125669 12773 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: No bootstrap required, opened a new log
I20260620 19:19:05.125733 12773 ts_tablet_manager.cc:1397] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 19:19:05.128048 12773 raft_consensus.cc:348] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.128247 12773 raft_consensus.cc:374] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:05.128286 12773 raft_consensus.cc:729] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4efd5ea8256434087cf85ef00e7d440, State: Initialized, Role: FOLLOWER
I20260620 19:19:05.128423 12773 consensus_queue.cc:260] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.128641 12777 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Bootstrap starting.
I20260620 19:19:05.128674 12773 ts_tablet_manager.cc:1428] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:19:05.128724 12482 heartbeater.cc:503] Master 127.1.1.254:37123 was elected leader, sending a full tablet report...
I20260620 19:19:05.129526 12777 tablet_bootstrap.cc:654] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:05.130035 12777 log.cc:824] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:05.130843 12777 tablet_bootstrap.cc:492] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: No bootstrap required, opened a new log
I20260620 19:19:05.130913 12777 ts_tablet_manager.cc:1397] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:05.133148 12777 raft_consensus.cc:348] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.133344 12777 raft_consensus.cc:374] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:05.133422 12777 raft_consensus.cc:729] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbb1f8b3b9a64613b1959ec66ae06962, State: Initialized, Role: FOLLOWER
I20260620 19:19:05.133579 12777 consensus_queue.cc:260] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.133879 12777 ts_tablet_manager.cc:1428] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:19:05.134025 12620 heartbeater.cc:503] Master 127.1.1.254:37123 was elected leader, sending a full tablet report...
Jun 20 19:19:05 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
W20260620 19:19:05.315264 12621 tablet.cc:2369] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:19:05.323376 12755 tablet.cc:2369] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:19:05.334828 12483 tablet.cc:2369] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:19:05 dist-test-slave-mwkb krb5kdc[11794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983144, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:05.557212 12778 raft_consensus.cc:482] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:05.557365 12778 raft_consensus.cc:504] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.557735 12778 leader_election.cc:288] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c7ff4e750b8456aac9cc010dfdf4cf0 (127.1.1.195:32985), cbb1f8b3b9a64613b1959ec66ae06962 (127.1.1.194:36015)
I20260620 19:19:05.561826 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.557973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:46649 (local address 127.1.1.194:36015)
0620 19:19:05.558062 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.558065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.558308 (+ 243us) server_negotiation.cc:407] Connection header received
0620 19:19:05.558434 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.558435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.558444 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.558474 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:05.558696 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.559443 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.560941 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.561472 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.561518 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.561528 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.561725 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.561758 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 19:19:05.561841 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.558228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:46065 (local address 127.1.1.195:32985)
0620 19:19:05.558269 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.558271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.558416 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:19:05.558512 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.558513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.558523 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.558550 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:05.558760 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.559495 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.560922 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.561486 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.561531 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.561540 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.561743 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.561776 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:19:05.562062 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.558109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:32985 (local address 127.1.1.193:46065)
0620 19:19:05.558357 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.558366 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.558440 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.558609 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.558610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.558612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:05.558691 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.558693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.559607 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.559608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.560809 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.560811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.561595 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.561596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.561671 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.561674 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.561959 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.561995 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":65,"threads_started":1}
I20260620 19:19:05.562438 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce042fa402441249a1643ea9823be92" candidate_uuid: "f4efd5ea8256434087cf85ef00e7d440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" is_pre_election: true
I20260620 19:19:05.562585 12708 raft_consensus.cc:2454] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4efd5ea8256434087cf85ef00e7d440 in term 0.
I20260620 19:19:05.562798 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.557912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36015 (local address 127.1.1.193:46649)
0620 19:19:05.558227 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.558238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:05.558356 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:05.558540 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:05.558541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:05.558543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:05.558621 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.558623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.559546 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.559548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.560829 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:05.560842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.561585 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.561585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:05.561652 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.561655 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:05.562228 (+ 573us) client_negotiation.cc:240] Negotiation successful
0620 19:19:05.562269 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":91,"threads_started":1}
I20260620 19:19:05.562803 12370 leader_election.cc:302] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c7ff4e750b8456aac9cc010dfdf4cf0, f4efd5ea8256434087cf85ef00e7d440; no voters:
I20260620 19:19:05.563133 12778 raft_consensus.cc:2781] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:19:05.563172 12778 raft_consensus.cc:482] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:19:05.563191 12778 raft_consensus.cc:3037] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:05.563239 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce042fa402441249a1643ea9823be92" candidate_uuid: "f4efd5ea8256434087cf85ef00e7d440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" is_pre_election: true
I20260620 19:19:05.563366 12570 raft_consensus.cc:2454] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4efd5ea8256434087cf85ef00e7d440 in term 0.
I20260620 19:19:05.564052 12778 raft_consensus.cc:504] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.564172 12778 leader_election.cc:288] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [CANDIDATE]: Term 1 election: Requested vote from peers 9c7ff4e750b8456aac9cc010dfdf4cf0 (127.1.1.195:32985), cbb1f8b3b9a64613b1959ec66ae06962 (127.1.1.194:36015)
I20260620 19:19:05.564383 12570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce042fa402441249a1643ea9823be92" candidate_uuid: "f4efd5ea8256434087cf85ef00e7d440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbb1f8b3b9a64613b1959ec66ae06962"
I20260620 19:19:05.564404 12708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fce042fa402441249a1643ea9823be92" candidate_uuid: "f4efd5ea8256434087cf85ef00e7d440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0"
I20260620 19:19:05.564445 12570 raft_consensus.cc:3037] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:05.564476 12708 raft_consensus.cc:3037] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:05.565205 12570 raft_consensus.cc:2454] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4efd5ea8256434087cf85ef00e7d440 in term 1.
I20260620 19:19:05.565358 12708 raft_consensus.cc:2454] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4efd5ea8256434087cf85ef00e7d440 in term 1.
I20260620 19:19:05.565389 12369 leader_election.cc:302] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbb1f8b3b9a64613b1959ec66ae06962, f4efd5ea8256434087cf85ef00e7d440; no voters:
I20260620 19:19:05.565457 12778 raft_consensus.cc:2781] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:05.565555 12778 raft_consensus.cc:686] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 1 LEADER]: Becoming Leader. State: Replica: f4efd5ea8256434087cf85ef00e7d440, State: Running, Role: LEADER
I20260620 19:19:05.565630 12778 consensus_queue.cc:237] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } }
I20260620 19:19:05.566322 12270 catalog_manager.cc:5495] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4efd5ea8256434087cf85ef00e7d440 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "f4efd5ea8256434087cf85ef00e7d440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4efd5ea8256434087cf85ef00e7d440" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34887 } health_report { overall_health: HEALTHY } } }
I20260620 19:19:05.654716 12796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.652200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.1.1.193:34887)
0620 19:19:05.652324 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:05.652326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:05.652410 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:19:05.652491 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:05.652492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:05.652519 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:05.652603 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:19:05.652743 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.653212 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.653959 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:05.654148 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:05.654232 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:05.654392 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:19:05.654491 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:19:05.654583 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:05.654605 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 19:19:05.654623 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":82,"threads_started":1}
I20260620 19:19:05.656343 12570 raft_consensus.cc:1261] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962 [term 1 FOLLOWER]: Refusing update from remote peer f4efd5ea8256434087cf85ef00e7d440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:05.656351 12708 raft_consensus.cc:1261] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 1 FOLLOWER]: Refusing update from remote peer f4efd5ea8256434087cf85ef00e7d440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:05.656648 12785 consensus_queue.cc:1035] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c7ff4e750b8456aac9cc010dfdf4cf0" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:05.656790 12778 consensus_queue.cc:1035] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbb1f8b3b9a64613b1959ec66ae06962" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:05.659783 12801 mvcc.cc:205] Tried to move back new op lower bound from 7299002964605894656 to 7299002964237271040. Current Snapshot: MvccSnapshot[applied={T|T < 7299002964605894656}]
I20260620 19:19:05.660766 12800 mvcc.cc:205] Tried to move back new op lower bound from 7299002964605894656 to 7299002964237271040. Current Snapshot: MvccSnapshot[applied={T|T < 7299002964605894656}]
I20260620 19:19:05.661340 12797 mvcc.cc:205] Tried to move back new op lower bound from 7299002964605894656 to 7299002964237271040. Current Snapshot: MvccSnapshot[applied={T|T < 7299002964605894656 or (T in {7299002964605894656})}]
I20260620 19:19:05.661787 12269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:05.661862 12269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:05.662533 12269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ad6db7ad3b14bde818902200c50dec2: Sending DeleteTablet for 3 replicas of tablet fce042fa402441249a1643ea9823be92
I20260620 19:19:05.662833 12670 tablet_service.cc:1514] Processing DeleteTablet for tablet fce042fa402441249a1643ea9823be92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:05 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:42650
I20260620 19:19:05.662835 12548 tablet_service.cc:1514] Processing DeleteTablet for tablet fce042fa402441249a1643ea9823be92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:05 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:37934
I20260620 19:19:05.663272 12416 tablet_service.cc:1514] Processing DeleteTablet for tablet fce042fa402441249a1643ea9823be92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:05 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:59708
I20260620 19:19:05.663503 12810 tablet_replica.cc:330] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440: stopping tablet replica
I20260620 19:19:05.663595 12810 raft_consensus.cc:2229] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:19:05.663586 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 12351
I20260620 19:19:05.663674 12810 raft_consensus.cc:2258] T fce042fa402441249a1643ea9823be92 P f4efd5ea8256434087cf85ef00e7d440 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:19:05.668582 12256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:34887 (error 108)
I20260620 19:19:05.668685 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 12488
I20260620 19:19:05.668699 12809 tablet_replica.cc:330] T fce042fa402441249a1643ea9823be92 P cbb1f8b3b9a64613b1959ec66ae06962: stopping tablet replica
I20260620 19:19:05.669438 12808 tablet_replica.cc:330] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: stopping tablet replica
I20260620 19:19:05.669525 12808 raft_consensus.cc:2229] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:05.669610 12808 raft_consensus.cc:2258] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:05.669984 12808 ts_tablet_manager.cc:1905] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:05.671662 12808 ts_tablet_manager.cc:1918] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:05.671742 12808 log.cc:1196] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/fce042fa402441249a1643ea9823be92
I20260620 19:19:05.672020 12808 ts_tablet_manager.cc:1939] T fce042fa402441249a1643ea9823be92 P 9c7ff4e750b8456aac9cc010dfdf4cf0: Deleting consensus metadata
I20260620 19:19:05.672161 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.672003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34887 (local address 127.0.0.1:59724)
0620 19:19:05.672084 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.672120 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":36,"threads_started":1}
W20260620 19:19:05.672317 12256 catalog_manager.cc:4584] TS f4efd5ea8256434087cf85ef00e7d440 (127.1.1.193:34887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fce042fa402441249a1643ea9823be92: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34887: connect: Connection refused (error 111)
I20260620 19:19:05.672442 12257 catalog_manager.cc:4841] TS 9c7ff4e750b8456aac9cc010dfdf4cf0 (127.1.1.195:32985): tablet fce042fa402441249a1643ea9823be92 (table test-table [id=582c2138d4374c8284594987829a7320]) successfully deleted
I20260620 19:19:05.674041 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 12625
I20260620 19:19:05.674600 12812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:05.673958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36015 (local address 127.0.0.1:37936)
0620 19:19:05.674546 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:05.674572 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.194:36015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572}
I20260620 19:19:05.678205 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 12238
I20260620 19:19:05.683144 1031 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:19:06.018321 1031 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:19:06.018420 1031 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:19:06.362193 1031 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:19:06.362 UTC [11824] LOG: received smart shutdown request
2026-06-20 19:19:06.365 UTC [11824] LOG: background worker "logical replication launcher" (PID 11833) exited with exit code 1
2026-06-20 19:19:06.365 UTC [11828] LOG: shutting down
2026-06-20 19:19:06.475 UTC [11824] LOG: database system is shut down
2026-06-20T19:19:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (41195 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:19:06 dist-test-slave-mwkb krb5kdc[12821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:19:06 dist-test-slave-mwkb krb5kdc[12821](info): set up 2 sockets
Jun 20 19:19:06 dist-test-slave-mwkb krb5kdc[12821](info): commencing operation
krb5kdc: starting...
W20260620 19:19:07.548722 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:19:07 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:19:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:19:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangeradmin_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangeradmin_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangerlookup_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangerlookup_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
I20260620 19:19:07.681473 1031 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNtb4aU/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/postgres -l logfile start
2026-06-20 19:19:09.789 UTC [12851] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:19:09.789 UTC [12851] LOG: listening on IPv4 address "127.1.1.212", port 58221
2026-06-20 19:19:09.792 UTC [12851] LOG: listening on Unix socket "/tmp/.s.PGSQL.58221"
2026-06-20 19:19:09.824 UTC [12854] LOG: database system was shut down at 2026-06-20 19:19:08 UTC
2026-06-20 19:19:09.827 UTC [12851] LOG: database system is ready to accept connections
I20260620 19:19:10.712767 1031 mini_postgres.cc:96] Postgres bound to 58221
2026-06-20 19:19:10.718 UTC [12862] FATAL: database "slave" does not exist
127.1.1.212:58221 - accepting connections
I20260620 19:19:10.718586 1031 mini_ranger.cc:161] Starting Ranger...
I20260620 19:19:10.731534 1031 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:19:11.115501 1031 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:19:11.115562 1031 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin
2026-06-20 19:19:11,362 [I] DB FLAVOR :POSTGRES
2026-06-20 19:19:11,363 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:19:11,363 [I] Checking connection..
2026-06-20 19:19:11,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:19:11,595 [I] Checking connection passed.
2026-06-20 19:19:11,595 [I] --------- Verifying version history table ---------
2026-06-20 19:19:11,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:19:11,805 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:19:11,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:19:12,011 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:19:12,012 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:19:12,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:19:12.246 UTC [12926] WARNING: there is no transaction in progress
2026-06-20 19:19:12,252 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:19:12,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:19:12,488 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:19:12,488 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:19:12,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:19:12,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:19:12,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mwkb.c.gcp-upstream.internal','N') ;"
2026-06-20 19:19:13,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:19:13,354 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:19:13,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:19:13,602 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:19:13,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:19:13,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:19:14,069 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:19:14,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:19:14.893 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:14.908 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:14.923 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:14.936 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:14.953 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:14.969 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.027 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.033 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.045 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.054 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.064 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.076 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.084 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.090 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.095 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15.101 UTC [13034] WARNING: there is no transaction in progress
2026-06-20 19:19:15,641 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:19:15,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:19:15,851 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:19:15,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:19:16,061 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:19:16,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:19:16,298 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:19:16,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:19:16,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:19:16,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mwkb.c.gcp-upstream.internal';"
2026-06-20 19:19:16,993 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:19:16,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:19:17,199 [I] DB_PATCHES have already been applied
I20260620 19:19:17.204793 1031 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNtb4aU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:19:17.204845 1031 mini_ranger.cc:193] Using host: 127.1.1.212
I20260620 19:19:17.207589 1031 mini_ranger.cc:227] Ranger admin URL: http://127.1.1.212:35935
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:19:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:19:18.245883 1031 mini_ranger.cc:160] Time spent starting Ranger: real 7.527s user 0.000s sys 0.007s
Jun 20, 2026 7:19:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:19:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 7:19:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:19:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:19:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:19:37.438537 1031 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.1.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.1.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangerkms_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.1.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/rangerkms_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.1.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.1.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.1.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/HTTP_127.1.1.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:19:37 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983177, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:19:37.521492 1031 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:19:37.529541 1031 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:19:38.314997 1031 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:19:38.315075 1031 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms
I20260620 19:19:38.315092 1031 mini_ranger_kms.cc:227] Using postgres at 127.1.1.212:58221
2026-06-20 19:19:38,472 [I] DB FLAVOR :POSTGRES
2026-06-20 19:19:38,472 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:19:38,472 [I] Checking connection
2026-06-20 19:19:38,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:19:38,712 [I] connection success
2026-06-20 19:19:38,712 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:19:38,713 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:19:38,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:19:38,927 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:19:38,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:19:38,927 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:19:38,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.1.212:58221/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:19:39,171 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:19:39.622192 1031 mini_ranger_kms.cc:310] Created kms service
I20260620 19:19:39.801417 1031 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:19:39.841187 1031 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:19:39 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983177, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:40.239802 1031 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:19:40.239912 1031 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNtb4aU/build/release/bin/postgresql.jar:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNtb4aU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:19:40.239931 1031 mini_ranger_kms.cc:240] Using host: 127.1.1.212
I20260620 19:19:40.243199 1031 mini_ranger_kms.cc:276] Ranger KMS PID: 13214
I20260620 19:19:40.243261 1031 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.1.212:41171
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:19:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:19:42.441253 1031 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.920s user 0.011s sys 0.130s
I20260620 19:19:42.441345 1031 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:19:42.441390 1031 mini_ranger_kms.cc:401] 127.1.1.212:41171/kms/v1/keys
Jun 20 19:19:42 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781983182, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:19:42 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983182, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
Jun 20 19:19:43 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983182, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.1.212@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
Jun 20 19:19:45 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:19:45.351089 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44517
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44517
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:45.428318 13265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:45.428521 13265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:45.428543 13265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:45.429800 13265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:45.429841 13265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:45.429855 13265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:45.429868 13265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:45.429880 13265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:45.431427 13265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.1.212:41171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44517
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:45.431655 13265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:45.431881 13265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:19:45.436812 13265 server_base.cc:1034] running on GCE node
W20260620 19:19:45.436954 13271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:45.436864 13270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:45.436877 13273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:45.437345 13265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:45.437645 13265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:45.438817 13265 hybrid_clock.cc:648] HybridClock initialized: now 1781983185438797 us; error 34 us; skew 500 ppm
Jun 20 19:19:45 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983185, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:45.445173 13265 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:45.445552 13265 webserver.cc:460] Webserver started at http://127.1.1.254:37933/ using document root <none> and password file <none>
I20260620 19:19:45.445747 13265 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:45.445798 13265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:45.445919 13265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:45 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983185, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:45.496049 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "1a87551286e2424bad949f835644504a"
format_stamp: "Formatted at 2026-06-20 19:19:45 on dist-test-slave-mwkb"
server_key: "db8a5c7edc13e8b712cd7a22c1720bb3"
server_key_iv: "5968e9fb5179683c11fa19a781fd3a67"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:45.496515 13265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "1a87551286e2424bad949f835644504a"
format_stamp: "Formatted at 2026-06-20 19:19:45 on dist-test-slave-mwkb"
server_key: "db8a5c7edc13e8b712cd7a22c1720bb3"
server_key_iv: "5968e9fb5179683c11fa19a781fd3a67"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:45.498344 13265 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 19:19:45.512217 13280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:45.512409 13265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:45.512511 13265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "1a87551286e2424bad949f835644504a"
format_stamp: "Formatted at 2026-06-20 19:19:45 on dist-test-slave-mwkb"
server_key: "db8a5c7edc13e8b712cd7a22c1720bb3"
server_key_iv: "5968e9fb5179683c11fa19a781fd3a67"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:45.512611 13265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:45.524472 13265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:45.526543 13265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:45.526685 13265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:45.531112 13265 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44517
I20260620 19:19:45.531150 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44517 every 8 connection(s)
I20260620 19:19:45.531476 13265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:45.532212 13333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:45.535318 13333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Bootstrap starting.
I20260620 19:19:45.535797 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13265
I20260620 19:19:45.535897 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:19:45.536024 13333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:45.536049 1031 external_mini_cluster.cc:1428] Setting key f1a07654f639c29d38e75008eb582199
I20260620 19:19:45.536705 13333 log.cc:824] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:45.537413 13333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: No bootstrap required, opened a new log
I20260620 19:19:45.539582 13333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:45.539733 13333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:45.539777 13333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87551286e2424bad949f835644504a, State: Initialized, Role: FOLLOWER
I20260620 19:19:45.539909 13333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:45.539973 13333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:45.539994 13333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:45.540040 13333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:45.540669 13333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:45.540771 13333 leader_election.cc:302] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a87551286e2424bad949f835644504a; no voters:
I20260620 19:19:45.540956 13333 leader_election.cc:288] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:19:45.541167 13339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:45.541281 13333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:45.541373 13339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 LEADER]: Becoming Leader. State: Replica: 1a87551286e2424bad949f835644504a, State: Running, Role: LEADER
I20260620 19:19:45.541507 13339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:45.542662 13338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a87551286e2424bad949f835644504a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } } }
I20260620 19:19:45.542747 13338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:45.542923 13340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a87551286e2424bad949f835644504a. Latest consensus state: current_term: 1 leader_uuid: "1a87551286e2424bad949f835644504a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } } }
I20260620 19:19:45.542984 13340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:45.543222 13352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:45.543673 13352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:19:45 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:45.544226 13333 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:19:45.550609 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:45.536478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.1.1.254:44517)
0620 19:19:45.536979 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:45.536988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:45.537019 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:19:45.537578 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:45.537586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:45.537645 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:45.537788 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:45.538577 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:45.539957 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:45.541054 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:45.541688 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:45.541798 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:45.544563 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:45.544567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:45.544567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:45.544577 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:45.548698 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:45.549192 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:45.549194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:45.549198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:45.549328 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:45.549572 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:45.549573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:45.549573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:45.549896 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:45.550081 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:45.550147 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:19:45.550304 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":84,"threads_started":1}
I20260620 19:19:45.553642 13352 catalog_manager.cc:1347] Generated new cluster ID: 70181c8e5f3042cd853b4d23b65bc2f7
I20260620 19:19:45.553686 13352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:45.587288 13352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:19:45.588261 13352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:45.607957 13352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Generated new TSK 0
I20260620 19:19:45.608271 13352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:19:46.315208 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:46.426378 13378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:46.426596 13378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:46.426631 13378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:46.427793 13378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:46.427850 13378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:46.427891 13378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:46.429373 13378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:46.429612 13378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:46.429838 13378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:46.434497 13386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.434500 13383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.434774 13384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:46.434935 13378 server_base.cc:1034] running on GCE node
I20260620 19:19:46.435119 13378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:46.435359 13378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:46.436519 13378 hybrid_clock.cc:648] HybridClock initialized: now 1781983186436404 us; error 144 us; skew 500 ppm
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:46.439126 13378 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:46.439601 13378 webserver.cc:460] Webserver started at http://127.1.1.193:41507/ using document root <none> and password file <none>
I20260620 19:19:46.439841 13378 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:46.439900 13378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:46.440032 13378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:46.452330 13378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "7ec13addcfa648cd94e31ba3830dd2a2"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "bc14f8b7c095ec9cbc9ee9b2592bed8c"
server_key_iv: "66d3f588279f0f46580732de1e9f18db"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.452728 13378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "7ec13addcfa648cd94e31ba3830dd2a2"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "bc14f8b7c095ec9cbc9ee9b2592bed8c"
server_key_iv: "66d3f588279f0f46580732de1e9f18db"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.460773 13378 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.002s
I20260620 19:19:46.473130 13393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.473335 13378 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:19:46.473459 13378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "7ec13addcfa648cd94e31ba3830dd2a2"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "bc14f8b7c095ec9cbc9ee9b2592bed8c"
server_key_iv: "66d3f588279f0f46580732de1e9f18db"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.473615 13378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:46.501237 13378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:46.503687 13378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:46.503829 13378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:46.504062 13378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:46.504379 13378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:46.504422 13378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.504452 13378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:46.504504 13378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.511111 13378 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:39607
I20260620 19:19:46.511500 13378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:19:46.517616 13508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:39607 every 8 connection(s)
I20260620 19:19:46.521286 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13378
I20260620 19:19:46.521385 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:19:46.521597 1031 external_mini_cluster.cc:1428] Setting key 963ed29deabfc6b696b4c3987301c7a6
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:46.529934 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.517886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.193:60183)
0620 19:19:46.518066 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:46.518112 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:46.518809 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:46.519103 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:46.519106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:46.519288 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:46.519474 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.519478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.520216 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.520218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.521005 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.521007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.521536 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.521537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.521637 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.522217 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:46.522235 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:46.523902 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:46.527285 (+ 3383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.527292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.527294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.527609 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.527859 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.527860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.527861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.527998 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.529233 (+ 1235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:46.529240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:46.529420 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 19:19:46.529537 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:19:46.529700 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260620 19:19:46.530423 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.518416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:60183 (local address 127.1.1.254:44517)
0620 19:19:46.518585 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:46.518587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:46.518599 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:19:46.518931 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:46.518932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:46.518947 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:46.519017 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:46.519581 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.520115 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.521161 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.521396 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.521464 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.524012 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:46.524013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:46.524013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:46.524020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:46.526668 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.527709 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.527711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.527711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.527780 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.528599 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.528602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.528603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.529043 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:46.529210 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:46.530214 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0620 19:19:46.530312 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260620 19:19:46.530776 13509 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:46.530881 13509 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:46.531104 13509 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:46.531677 13297 ts_manager.cc:194] Registered new tserver with Master: 7ec13addcfa648cd94e31ba3830dd2a2 (127.1.1.193:39607)
I20260620 19:19:46.532541 13297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:60183
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:19:46.557444 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 19:19:46.672366 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:46.672652 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:46.672731 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:46.674477 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:46.674532 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:46.674571 13515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:46.676450 13515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:46.676666 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:46.676952 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:46.683557 13523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.683810 13525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.684051 13522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:46.685109 13515 server_base.cc:1034] running on GCE node
I20260620 19:19:46.685313 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:46.685576 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:46.688876 13515 hybrid_clock.cc:648] HybridClock initialized: now 1781983186688847 us; error 37 us; skew 500 ppm
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:46.692905 13515 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:46.693711 13515 webserver.cc:460] Webserver started at http://127.1.1.194:43531/ using document root <none> and password file <none>
I20260620 19:19:46.693982 13515 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:46.694036 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:46.694168 13515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:46.717810 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "f15d50333f1c4dc8ad4f7030f6f46413"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "e21d1218e16560139cf725183e65e5b4"
server_key_iv: "d0f970310ccac750281e5bd501827452"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.719359 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "f15d50333f1c4dc8ad4f7030f6f46413"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "e21d1218e16560139cf725183e65e5b4"
server_key_iv: "d0f970310ccac750281e5bd501827452"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.722836 13515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:19:46.734385 13532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.734548 13515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:19:46.734628 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "f15d50333f1c4dc8ad4f7030f6f46413"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "e21d1218e16560139cf725183e65e5b4"
server_key_iv: "d0f970310ccac750281e5bd501827452"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.734695 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:46.745947 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:46.748059 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:46.748221 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:46.748435 13515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:46.748800 13515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:46.748880 13515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.748953 13515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:46.749010 13515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.755440 13515 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:35883
I20260620 19:19:46.755807 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:19:46.756160 13646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:35883 every 8 connection(s)
I20260620 19:19:46.759624 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13515
I20260620 19:19:46.759714 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:19:46.759874 1031 external_mini_cluster.cc:1428] Setting key c8373832cb4f4a39b6dd0f32144fcf9e
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:46.768829 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.756863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:54193 (local address 127.1.1.254:44517)
0620 19:19:46.756905 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:46.756907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:46.757124 (+ 217us) server_negotiation.cc:407] Connection header received
0620 19:19:46.757756 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:46.757758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:46.757769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:46.757800 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:46.758592 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.759415 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.763094 (+ 3679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.763402 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.763456 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.765754 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:46.765756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:46.765756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:46.765762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:46.767256 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.767773 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.767774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.767775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.767832 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.768111 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.768112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.768113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.768246 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:46.768293 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:46.768657 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 19:19:46.768741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:19:46.769605 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.756817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.194:54193)
0620 19:19:46.756976 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:46.757016 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:46.757636 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:46.758003 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:46.758007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:46.758229 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:46.758483 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.758488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.760143 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.760144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.762964 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.762966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.763554 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.763555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.763682 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.764223 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:46.764232 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:46.765662 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:46.767345 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.767350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.767352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.767666 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.767901 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.767902 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.767902 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.768028 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.768323 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:46.768328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:46.768482 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 19:19:46.769245 (+ 763us) client_negotiation.cc:240] Negotiation successful
0620 19:19:46.769397 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260620 19:19:46.770391 13647 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:46.770493 13647 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:46.770648 13647 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:46.771000 13297 ts_manager.cc:194] Registered new tserver with Master: f15d50333f1c4dc8ad4f7030f6f46413 (127.1.1.194:35883)
I20260620 19:19:46.771446 13297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:54193
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:19:46.794090 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:46.910815 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:46.911139 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:46.911221 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:46.912979 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:46.913115 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:46.913199 13652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:46.915475 13652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:46.915925 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:46.916263 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:46.924055 13658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.924378 13660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:46.925170 13657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:46.925170 13652 server_base.cc:1034] running on GCE node
I20260620 19:19:46.925500 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:46.925750 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:46.929018 13652 hybrid_clock.cc:648] HybridClock initialized: now 1781983186928989 us; error 36 us; skew 500 ppm
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:46.931453 13652 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:46.931939 13652 webserver.cc:460] Webserver started at http://127.1.1.195:42811/ using document root <none> and password file <none>
I20260620 19:19:46.932183 13652 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:46.932256 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:46.932380 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:46.946444 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "5dcd3b9f50e242e795fee06fe0dd78b4"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "a750a24c13cee43727a3143120cf7864"
server_key_iv: "8af30ef6bf5dfca544fe720eaef55ecb"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.946817 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "5dcd3b9f50e242e795fee06fe0dd78b4"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "a750a24c13cee43727a3143120cf7864"
server_key_iv: "8af30ef6bf5dfca544fe720eaef55ecb"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.948192 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:19:46.966022 13667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.966243 13652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:46.966333 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "5dcd3b9f50e242e795fee06fe0dd78b4"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "a750a24c13cee43727a3143120cf7864"
server_key_iv: "8af30ef6bf5dfca544fe720eaef55ecb"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:46.966434 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:46.975994 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:46.978335 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:46.978520 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:46.978783 13652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:46.979187 13652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:46.979280 13652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.979384 13652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:46.979449 13652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:46.986428 13652 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:37711
I20260620 19:19:46.986518 13780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:37711 every 8 connection(s)
I20260620 19:19:46.986896 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:19:46.990687 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13652
I20260620 19:19:46.990777 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:19:46.990950 1031 external_mini_cluster.cc:1428] Setting key 8d7a886639e4ce1d0d893e1b0ae5524e
Jun 20 19:19:46 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983186, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:46.997359 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.987661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:37997 (local address 127.1.1.254:44517)
0620 19:19:46.987706 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:46.987708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:46.987983 (+ 275us) server_negotiation.cc:407] Connection header received
0620 19:19:46.988573 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:46.988574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:46.988585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:46.988640 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:46.989308 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.990017 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.991494 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.991782 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.991820 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.994313 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:46.994314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:46.994314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:46.994321 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:46.995784 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.996265 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.996266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.996267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.996327 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:46.996607 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:46.996608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:46.996608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:46.996733 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:46.996799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:46.997191 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 19:19:46.997272 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:19:46.997871 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:46.987651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.195:37997)
0620 19:19:46.987819 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:46.987864 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:46.988469 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:46.988721 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:46.988725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:46.988936 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:46.989221 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.989227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.990121 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.990122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.991383 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:46.991386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:46.991898 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:46.991899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:46.992027 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:46.992533 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:46.992541 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:46.994199 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:46.995875 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.995881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.995884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.996169 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.996400 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:46.996401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:46.996401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:46.996534 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:46.996804 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:46.996809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:46.996973 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 19:19:46.997510 (+ 537us) client_negotiation.cc:240] Negotiation successful
0620 19:19:46.997660 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260620 19:19:46.998349 13781 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:46.998445 13781 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:46.998622 13781 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:46.999010 13297 ts_manager.cc:194] Registered new tserver with Master: 5dcd3b9f50e242e795fee06fe0dd78b4 (127.1.1.195:37711)
I20260620 19:19:46.999486 13297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:37997
I20260620 19:19:47.002437 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:19:47.003158 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13378
I20260620 19:19:47.009800 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13515
I20260620 19:19:47.029938 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13652
I20260620 19:19:47.036661 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13265
I20260620 19:19:47.042081 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44517
--webserver_interface=127.1.1.254
--webserver_port=37933
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44517
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:47.141993 13784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:47.142180 13784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:47.142201 13784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:47.143332 13784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:47.143374 13784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:47.143388 13784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:47.143400 13784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:47.143412 13784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:47.145011 13784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44517
--ranger_config_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=37933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:47.145275 13784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:47.145496 13784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:47.150132 13789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.150235 13790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.150122 13792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:47.150390 13784 server_base.cc:1034] running on GCE node
I20260620 19:19:47.150607 13784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:47.150902 13784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:47.152060 13784 hybrid_clock.cc:648] HybridClock initialized: now 1781983187152048 us; error 26 us; skew 500 ppm
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:47.154529 13784 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:47.155015 13784 webserver.cc:460] Webserver started at http://127.1.1.254:37933/ using document root <none> and password file <none>
I20260620 19:19:47.155256 13784 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:47.155305 13784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:47.199237 13784 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:47.200003 13800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.200168 13784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:47.200245 13784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "1a87551286e2424bad949f835644504a"
format_stamp: "Formatted at 2026-06-20 19:19:45 on dist-test-slave-mwkb"
server_key: "db8a5c7edc13e8b712cd7a22c1720bb3"
server_key_iv: "5968e9fb5179683c11fa19a781fd3a67"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:47.200601 13784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:47.216100 13784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:47.218071 13784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:47.218227 13784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:47.222905 13784 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44517
I20260620 19:19:47.222934 13852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44517 every 8 connection(s)
I20260620 19:19:47.223294 13784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:47.224805 13853 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:19:47.225711 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Bootstrap starting.
I20260620 19:19:47.228284 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13784
I20260620 19:19:47.228569 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:39607
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:44517
--webserver_port=41507
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:47.235845 13853 log.cc:824] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:47.238049 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:19:47.238345 13853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Bootstrap complete.
I20260620 19:19:47.241680 13853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:47.242015 13853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a87551286e2424bad949f835644504a, State: Initialized, Role: FOLLOWER
I20260620 19:19:47.242174 13853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:47.242239 13853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:47.242280 13853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:47.242316 13853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:19:47.243556 13853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:47.243664 13853 leader_election.cc:302] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a87551286e2424bad949f835644504a; no voters:
I20260620 19:19:47.243813 13853 leader_election.cc:288] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:19:47.243918 13857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:19:47.244048 13853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:47.244130 13857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [term 2 LEADER]: Becoming Leader. State: Replica: 1a87551286e2424bad949f835644504a, State: Running, Role: LEADER
I20260620 19:19:47.244266 13857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } }
I20260620 19:19:47.244570 13857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a87551286e2424bad949f835644504a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } } }
I20260620 19:19:47.244690 13857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:47.244849 13857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a87551286e2424bad949f835644504a. Latest consensus state: current_term: 2 leader_uuid: "1a87551286e2424bad949f835644504a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a87551286e2424bad949f835644504a" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44517 } } }
I20260620 19:19:47.244987 13857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:47.245131 13866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:47.245492 13853 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:19:47.245667 13866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:47.245822 13866 catalog_manager.cc:1259] Loaded cluster ID: 70181c8e5f3042cd853b4d23b65bc2f7
I20260620 19:19:47.245883 13866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:47.250010 13866 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:47.250385 13866 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Loaded TSK: 0
I20260620 19:19:47.250706 13866 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:19:47.338654 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:47.338912 13855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:47.338960 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:47.340579 13855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:47.340632 13855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:47.340672 13855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:47.342677 13855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:39607
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=41507
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:47.342901 13855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:47.343149 13855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:47.349582 13891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.349817 13893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.350219 13890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:47.351238 13855 server_base.cc:1034] running on GCE node
I20260620 19:19:47.353413 13855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:47.353683 13855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:47.354857 13855 hybrid_clock.cc:648] HybridClock initialized: now 1781983187354822 us; error 48 us; skew 500 ppm
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:47.357856 13855 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:47.358392 13855 webserver.cc:460] Webserver started at http://127.1.1.193:41507/ using document root <none> and password file <none>
I20260620 19:19:47.358671 13855 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:47.358757 13855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:47.381470 13855 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:19:47.385277 13901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.385450 13855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 19:19:47.385532 13855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "7ec13addcfa648cd94e31ba3830dd2a2"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "bc14f8b7c095ec9cbc9ee9b2592bed8c"
server_key_iv: "66d3f588279f0f46580732de1e9f18db"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:47.385983 13855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:47.408252 13855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:47.410393 13855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:47.410547 13855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:47.410755 13855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:47.411072 13855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:47.411120 13855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.411149 13855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:47.411170 13855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.417711 13855 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:39607
I20260620 19:19:47.418138 13855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:19:47.422751 14014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:39607 every 8 connection(s)
I20260620 19:19:47.424065 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 13855
I20260620 19:19:47.424338 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:35883
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:44517
--webserver_port=43531
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:47.432387 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.418779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.193:40235)
0620 19:19:47.421363 (+ 2584us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:47.421402 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:47.421985 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:47.422592 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:47.422597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:47.422794 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:47.423393 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.423399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.425029 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.425030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.426230 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.426233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.427033 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.427034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.427140 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.427606 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:47.427614 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:47.429025 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:47.430947 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.430952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.430954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.431182 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.431406 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.431407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.431407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.431517 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.431892 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:47.431896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:47.432017 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 19:19:47.432096 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:19:47.432213 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2547,"thread_start_us":71,"threads_started":1}
I20260620 19:19:47.433117 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:40235 (local address 127.1.1.254:44517)
0620 19:19:47.421195 (+ 2331us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:47.421204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:47.421801 (+ 597us) server_negotiation.cc:407] Connection header received
0620 19:19:47.422321 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:47.422328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:47.422389 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:47.422520 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:47.423542 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.424915 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.426331 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.426859 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.426940 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.429156 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:47.429160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:47.429161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:47.429170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:47.430863 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.431261 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.431264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.431264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.431349 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.431575 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.431576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.431576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.431806 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:47.431868 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:47.432766 (+ 898us) server_negotiation.cc:299] Negotiation successful
0620 19:19:47.432867 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2279,"thread_start_us":81,"threads_started":1}
I20260620 19:19:47.434159 14015 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:47.434252 14015 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.434430 14015 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.435231 14015 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44517 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:19:47.435299 14015 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.435361 14015 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:47.435918 14015 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.436008 14015 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.436437 14015 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:19:47.530874 14019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:47.531126 14019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:47.531157 14019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:47.532639 14019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:47.532708 14019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:47.532745 14019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:47.534773 14019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:35883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=43531
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:47.534991 14019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:47.535235 14019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:47.541797 14027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.542794 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:47.543581 14019 server_base.cc:1034] running on GCE node
W20260620 19:19:47.543810 14025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:47.544492 14019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:47.544752 14019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:47.545917 14019 hybrid_clock.cc:648] HybridClock initialized: now 1781983187545894 us; error 37 us; skew 500 ppm
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:47.549044 14019 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:47.549659 14019 webserver.cc:460] Webserver started at http://127.1.1.194:43531/ using document root <none> and password file <none>
I20260620 19:19:47.549981 14019 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:47.550102 14019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:47.568549 14019 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:19:47.572880 14035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.573161 14019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 19:19:47.573302 14019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "f15d50333f1c4dc8ad4f7030f6f46413"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "e21d1218e16560139cf725183e65e5b4"
server_key_iv: "d0f970310ccac750281e5bd501827452"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:47.573854 14019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:47.630064 14019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:47.632496 14019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:47.632673 14019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:47.632958 14019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:47.633364 14019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:47.633410 14019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.633445 14019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:47.633504 14019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.639765 14019 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:35883
I20260620 19:19:47.640223 14019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:19:47.641665 14148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:35883 every 8 connection(s)
I20260620 19:19:47.645691 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14019
I20260620 19:19:47.645969 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.1.212:41171/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:37711
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:44517
--webserver_port=42811
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:47.651998 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.642178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.194:47981)
0620 19:19:47.642670 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:47.642710 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:47.643354 (+ 644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:47.643711 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:47.643716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:47.643947 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:47.644209 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.644216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.645294 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.645295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.646322 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.646324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.646724 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.646725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.646837 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.647284 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:47.647292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:47.648586 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:47.650434 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.650440 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.650442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.650726 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.650939 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.650939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.650940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.651059 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.651327 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:47.651333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:47.651492 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 19:19:47.651591 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:19:47.651749 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":71,"threads_started":1}
I20260620 19:19:47.652554 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.642426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:47981 (local address 127.1.1.254:44517)
0620 19:19:47.642477 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:47.642479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:47.642791 (+ 312us) server_negotiation.cc:407] Connection header received
0620 19:19:47.643482 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:47.643484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:47.643501 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:47.643534 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:47.644337 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.645128 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.646402 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.646619 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.646699 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.648673 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:47.648674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:47.648675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:47.648683 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:47.650332 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.650821 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.650823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.650823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.650875 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.651126 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.651127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.651127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.651250 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:47.651304 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:47.652334 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0620 19:19:47.652431 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:19:47.652844 14149 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:47.652926 14149 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.653136 14149 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.653759 14149 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44517 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:19:47.653844 14149 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.653914 14149 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:47.654386 14149 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.654474 14149 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.654778 14149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:19:47.738556 14152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:47.738806 14152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:47.738843 14152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:47.740473 14152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:47.740527 14152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:47.740563 14152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:47.742630 14152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.1.212:41171/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:37711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=42811
--tserver_master_addrs=127.1.1.254:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:47.742842 14152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:47.743062 14152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:47.753639 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.753639 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:47.753634 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:47.753904 14152 server_base.cc:1034] running on GCE node
I20260620 19:19:47.754104 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:47.754387 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:47.755582 14152 hybrid_clock.cc:648] HybridClock initialized: now 1781983187755565 us; error 32 us; skew 500 ppm
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:47.758553 14152 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:47.758996 14152 webserver.cc:460] Webserver started at http://127.1.1.195:42811/ using document root <none> and password file <none>
I20260620 19:19:47.759253 14152 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:47.759320 14152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:47.778004 14152 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 19:19:47.778972 14169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.779165 14152 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:47.779265 14152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "5dcd3b9f50e242e795fee06fe0dd78b4"
format_stamp: "Formatted at 2026-06-20 19:19:46 on dist-test-slave-mwkb"
server_key: "a750a24c13cee43727a3143120cf7864"
server_key_iv: "8af30ef6bf5dfca544fe720eaef55ecb"
server_key_version: "kuduclusterkey@0"
I20260620 19:19:47.779721 14152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:47.807582 14152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:47.809522 14152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:47.809679 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:47.809885 14152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:47.810194 14152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:47.810248 14152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.810281 14152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:47.810307 14152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:47.815698 14152 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:37711
I20260620 19:19:47.815840 14282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:37711 every 8 connection(s)
I20260620 19:19:47.816109 14152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:19:47.822606 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14152
Jun 20 19:19:47 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983187, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:47.826203 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.816822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:41629 (local address 127.1.1.254:44517)
0620 19:19:47.816897 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:47.816900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:47.817115 (+ 215us) server_negotiation.cc:407] Connection header received
0620 19:19:47.817667 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:47.817668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:47.817679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:47.817731 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:47.818265 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.819001 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.820406 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.820700 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.820787 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.823213 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:47.823214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:47.823215 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:47.823222 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:47.824691 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.825236 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.825237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.825237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.825298 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.825589 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.825590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.825591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.825700 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:47.825775 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:47.826042 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 19:19:47.826120 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:19:47.826362 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.816807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44517 (local address 127.1.1.195:41629)
0620 19:19:47.816954 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:47.816992 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:47.817562 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:47.817794 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:47.817798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:47.817987 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:47.818163 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.818167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.819112 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.819113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.820299 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:47.820302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.820827 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.820829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:47.820966 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.821483 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:47.821492 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:47.823070 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:47.824793 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.824800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.824802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.825151 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.825375 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:47.825376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:47.825377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:47.825515 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:47.825779 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:47.825785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:47.825914 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:19:47.826009 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 19:19:47.826139 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 19:19:47.826900 14283 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44517
I20260620 19:19:47.826987 14283 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.827145 14283 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.827809 14283 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44517 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:19:47.827878 14283 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.827950 14283 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:47.828349 14283 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:47.828430 14283 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
W20260620 19:19:47.828828 14283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:19:47.832401 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:47.823171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.1.1.254:44517)
0620 19:19:47.823275 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:47.823277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:47.824025 (+ 748us) server_negotiation.cc:407] Connection header received
0620 19:19:47.824097 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:47.824098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:47.824111 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:47.824184 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:47.825541 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.826405 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.827455 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:47.827755 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:47.827821 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:47.829776 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:47.829777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:47.829777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:47.829784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:47.831302 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.831662 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.831664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.831664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.831723 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:47.831915 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:47.831916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:47.831917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:47.832038 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:47.832189 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:47.832229 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:19:47.832305 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":59,"threads_started":1}
Jun 20 19:19:48 dist-test-slave-mwkb krb5kdc[12821](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781983188, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:48.436614 14015 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:48.436753 14015 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:48.437134 13817 ts_manager.cc:194] Registered new tserver with Master: 7ec13addcfa648cd94e31ba3830dd2a2 (127.1.1.193:39607)
I20260620 19:19:48.438124 13817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:40235
Jun 20 19:19:48 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983188, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:48.654991 14149 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:48.655122 14149 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:48.655391 13817 ts_manager.cc:194] Registered new tserver with Master: f15d50333f1c4dc8ad4f7030f6f46413 (127.1.1.194:35883)
I20260620 19:19:48.655857 13817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:47981
I20260620 19:19:48.829006 14283 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:48.829164 14283 heartbeater.cc:511] Master 127.1.1.254:44517 requested a full tablet report, sending...
I20260620 19:19:48.829442 13817 ts_manager.cc:194] Registered new tserver with Master: 5dcd3b9f50e242e795fee06fe0dd78b4 (127.1.1.195:37711)
I20260620 19:19:48.829733 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:19:48.829931 13817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:41629
I20260620 19:19:48.839088 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.830631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.1.1.254:44517)
0620 19:19:48.830898 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:48.830900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:48.831055 (+ 155us) server_negotiation.cc:407] Connection header received
0620 19:19:48.831077 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:48.831077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:48.831089 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:48.831143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:48.831889 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.832598 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.833595 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.833802 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.833878 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.836003 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:48.836004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:48.836004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:48.836011 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:48.837729 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:48.838163 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:48.838165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:48.838165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:48.838237 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:48.838435 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:48.838435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:48.838436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:48.838578 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:48.838662 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:48.838865 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:19:48.838962 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":61,"threads_started":1}
Jun 20 19:19:48 dist-test-slave-mwkb krb5kdc[12821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983188, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for HTTP/127.1.1.212@KRBTEST.COM
I20260620 19:19:48.841341 13817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:19:48.843530 13817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:19:48.859987 14312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39607 (local address 127.0.0.1:42766)
0620 19:19:48.853497 (+ 2946us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:48.853506 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:48.853547 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:48.853907 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:48.853909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:48.853911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:48.853967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.853968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.855514 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.855514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.858837 (+ 3323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.858839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.859738 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.859739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.859802 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.859813 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:19:48.859870 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:19:48.859912 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2934,"thread_start_us":97,"threads_started":1}
I20260620 19:19:48.860146 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:37711 (local address 127.0.0.1:52412)
0620 19:19:48.853805 (+ 3033us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:48.853814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:48.853847 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:48.856802 (+ 2955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:48.856802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:48.856804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:48.856839 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.856840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.857888 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.857888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.859022 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.859026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.859917 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.859918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.859974 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.859978 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:19:48.860021 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:19:48.860060 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3021,"thread_start_us":61,"threads_started":1}
I20260620 19:19:48.860334 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.1.1.193:39607)
0620 19:19:48.853443 (+ 2811us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:48.853458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:48.853595 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:19:48.853629 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:48.853631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:48.853647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:48.853709 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:48.854034 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.855048 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.858913 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.859599 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.859659 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.859708 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:48.860155 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 19:19:48.860264 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2778,"thread_start_us":48,"threads_started":1}
I20260620 19:19:48.860348 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.1.1.195:37711)
0620 19:19:48.856596 (+ 5664us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:48.856600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:48.856611 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:19:48.856640 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:48.856642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:48.856655 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:48.856727 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:48.856901 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.857796 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.859140 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.859778 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.859841 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.859884 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:48.860236 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 19:19:48.860282 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5632,"thread_start_us":45,"threads_started":1}
I20260620 19:19:48.861222 14217 tablet_service.cc:1467] Processing CreateTablet for tablet cc329ac1e7804107a6aad1d5535202ab (DEFAULT_TABLE table=test-table [id=b64e3abc094a4d0daac55971c818617b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:48.861171 13941 tablet_service.cc:1467] Processing CreateTablet for tablet cc329ac1e7804107a6aad1d5535202ab (DEFAULT_TABLE table=test-table [id=b64e3abc094a4d0daac55971c818617b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:48.861541 13941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc329ac1e7804107a6aad1d5535202ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:48.861539 14217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc329ac1e7804107a6aad1d5535202ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:48.863610 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:35883 (local address 127.0.0.1:50622)
0620 19:19:48.853198 (+ 2649us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:48.853254 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:48.853319 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:48.855398 (+ 2079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:48.855399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:48.855401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:48.855442 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.855443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.857230 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.857231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.858271 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:48.858274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.863372 (+ 5098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.863373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:48.863452 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.863456 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:19:48.863505 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:19:48.863544 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2620,"thread_start_us":82,"threads_started":1}
I20260620 19:19:48.863832 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:48.850802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.1.1.194:35883)
0620 19:19:48.855189 (+ 4387us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:48.855193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:48.855201 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:48.855230 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:48.855232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:48.855246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:48.855303 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:48.856202 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.857146 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.862386 (+ 5240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:48.863114 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:48.863184 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:48.863239 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:48.863710 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 19:19:48.863760 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4355,"thread_start_us":57,"threads_started":1}
I20260620 19:19:48.864419 14083 tablet_service.cc:1467] Processing CreateTablet for tablet cc329ac1e7804107a6aad1d5535202ab (DEFAULT_TABLE table=test-table [id=b64e3abc094a4d0daac55971c818617b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:48.864753 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc329ac1e7804107a6aad1d5535202ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:48.868109 14319 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Bootstrap starting.
I20260620 19:19:48.868758 14319 tablet_bootstrap.cc:654] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:48.869143 14319 log.cc:824] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:48.869935 14317 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Bootstrap starting.
I20260620 19:19:48.870481 14317 tablet_bootstrap.cc:654] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:48.870795 14317 log.cc:824] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:48.871120 14318 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Bootstrap starting.
I20260620 19:19:48.871730 14318 tablet_bootstrap.cc:654] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:48.872059 14318 log.cc:824] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:48.872830 14317 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: No bootstrap required, opened a new log
I20260620 19:19:48.872903 14317 ts_tablet_manager.cc:1397] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:19:48.873735 14318 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: No bootstrap required, opened a new log
I20260620 19:19:48.874156 14318 ts_tablet_manager.cc:1397] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:19:48.875310 14317 raft_consensus.cc:348] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.875479 14317 raft_consensus.cc:374] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:48.875517 14317 raft_consensus.cc:729] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec13addcfa648cd94e31ba3830dd2a2, State: Initialized, Role: FOLLOWER
I20260620 19:19:48.875626 14317 consensus_queue.cc:260] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.875850 14317 ts_tablet_manager.cc:1428] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:19:48.876004 14015 heartbeater.cc:503] Master 127.1.1.254:44517 was elected leader, sending a full tablet report...
I20260620 19:19:48.877278 14319 tablet_bootstrap.cc:492] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: No bootstrap required, opened a new log
I20260620 19:19:48.877357 14319 ts_tablet_manager.cc:1397] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 19:19:48.879585 14319 raft_consensus.cc:348] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.879760 14319 raft_consensus.cc:374] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:48.879806 14319 raft_consensus.cc:729] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15d50333f1c4dc8ad4f7030f6f46413, State: Initialized, Role: FOLLOWER
I20260620 19:19:48.879925 14319 consensus_queue.cc:260] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.880174 14319 ts_tablet_manager.cc:1428] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:19:48.880182 14149 heartbeater.cc:503] Master 127.1.1.254:44517 was elected leader, sending a full tablet report...
I20260620 19:19:48.880875 14318 raft_consensus.cc:348] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.881026 14318 raft_consensus.cc:374] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:48.881084 14318 raft_consensus.cc:729] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcd3b9f50e242e795fee06fe0dd78b4, State: Initialized, Role: FOLLOWER
I20260620 19:19:48.881191 14318 consensus_queue.cc:260] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:48.881456 14318 ts_tablet_manager.cc:1428] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260620 19:19:48.881583 14283 heartbeater.cc:503] Master 127.1.1.254:44517 was elected leader, sending a full tablet report...
W20260620 19:19:48.893618 14150 tablet.cc:2369] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:19:48.923831 14016 tablet.cc:2369] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:19:49.043390 14323 raft_consensus.cc:482] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:49.043531 14323 raft_consensus.cc:504] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:49.043923 14323 leader_election.cc:288] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f15d50333f1c4dc8ad4f7030f6f46413 (127.1.1.194:35883), 5dcd3b9f50e242e795fee06fe0dd78b4 (127.1.1.195:37711)
I20260620 19:19:49.048305 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.044818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:48719 (local address 127.1.1.195:37711)
0620 19:19:49.044855 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:49.044857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:49.045163 (+ 306us) server_negotiation.cc:407] Connection header received
0620 19:19:49.045250 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:49.045251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:49.045262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:49.045290 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:49.045466 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.046081 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.047385 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.047925 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.047972 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:49.047983 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:49.048195 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:19:49.048233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:19:49.048964 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.044416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:37711 (local address 127.1.1.193:48719)
0620 19:19:49.045050 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:49.045058 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:49.045183 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:49.045344 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:49.045345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:49.045346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:49.045403 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:49.045405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.046157 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.046157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:49.047286 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:49.047289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.048051 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.048052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:49.048118 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:49.048121 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:49.048819 (+ 698us) client_negotiation.cc:240] Negotiation successful
0620 19:19:49.048875 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260620 19:19:49.049407 14237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc329ac1e7804107a6aad1d5535202ab" candidate_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" is_pre_election: true
I20260620 19:19:49.049561 14237 raft_consensus.cc:2454] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec13addcfa648cd94e31ba3830dd2a2 in term 0.
I20260620 19:19:49.049767 13904 leader_election.cc:302] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcd3b9f50e242e795fee06fe0dd78b4, 7ec13addcfa648cd94e31ba3830dd2a2; no voters:
I20260620 19:19:49.049880 14323 raft_consensus.cc:2781] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:19:49.049917 14323 raft_consensus.cc:482] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:19:49.049939 14323 raft_consensus.cc:3037] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:49.050685 14323 raft_consensus.cc:504] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:49.050799 14323 leader_election.cc:288] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [CANDIDATE]: Term 1 election: Requested vote from peers f15d50333f1c4dc8ad4f7030f6f46413 (127.1.1.194:35883), 5dcd3b9f50e242e795fee06fe0dd78b4 (127.1.1.195:37711)
I20260620 19:19:49.050971 14237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc329ac1e7804107a6aad1d5535202ab" candidate_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4"
I20260620 19:19:49.051033 14237 raft_consensus.cc:3037] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:49.051780 14237 raft_consensus.cc:2454] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec13addcfa648cd94e31ba3830dd2a2 in term 1.
I20260620 19:19:49.051913 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.044743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:40825 (local address 127.1.1.194:35883)
0620 19:19:49.044791 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:49.044793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:49.044946 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:19:49.045026 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:49.045028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:49.045042 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:49.045110 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:49.045301 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.046016 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.050610 (+ 4594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.051087 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.051131 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:49.051140 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:49.051810 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 19:19:49.051853 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:19:49.051935 13904 leader_election.cc:302] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dcd3b9f50e242e795fee06fe0dd78b4, 7ec13addcfa648cd94e31ba3830dd2a2; no voters:
I20260620 19:19:49.052007 14323 raft_consensus.cc:2781] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:49.052120 14323 raft_consensus.cc:686] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec13addcfa648cd94e31ba3830dd2a2, State: Running, Role: LEADER
I20260620 19:19:49.052202 14323 consensus_queue.cc:237] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } }
I20260620 19:19:49.052342 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.044209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:35883 (local address 127.1.1.193:40825)
0620 19:19:49.044890 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:49.044901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:49.044964 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:49.045156 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:49.045157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:49.045159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:49.045234 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:49.045236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.046104 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.046105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:49.050503 (+ 4398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:49.050506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.051670 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.051671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:49.051736 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:49.051739 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:49.052139 (+ 400us) client_negotiation.cc:240] Negotiation successful
0620 19:19:49.052212 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":3070,"thread_start_us":99,"threads_started":1}
I20260620 19:19:49.052788 14103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc329ac1e7804107a6aad1d5535202ab" candidate_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" is_pre_election: true
I20260620 19:19:49.052917 14103 raft_consensus.cc:2454] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ec13addcfa648cd94e31ba3830dd2a2 in term 0.
I20260620 19:19:49.053013 14102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc329ac1e7804107a6aad1d5535202ab" candidate_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15d50333f1c4dc8ad4f7030f6f46413"
I20260620 19:19:49.053085 14102 raft_consensus.cc:3037] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:49.053922 14102 raft_consensus.cc:2454] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ec13addcfa648cd94e31ba3830dd2a2 in term 1.
I20260620 19:19:49.054122 13817 catalog_manager.cc:5495] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ec13addcfa648cd94e31ba3830dd2a2 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec13addcfa648cd94e31ba3830dd2a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 } health_report { overall_health: UNKNOWN } } }
W20260620 19:19:49.067463 14284 tablet.cc:2369] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:19:49.086678 14313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.084204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.1.1.193:39607)
0620 19:19:49.084327 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:49.084329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:49.084335 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:19:49.084364 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:49.084365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:49.084383 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:49.084466 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:19:49.084599 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.085176 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.085944 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:49.086143 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:49.086216 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:49.086357 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:19:49.086460 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:19:49.086553 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:49.086572 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 19:19:49.086592 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 19:19:49.088428 14237 raft_consensus.cc:1261] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 1 FOLLOWER]: Refusing update from remote peer 7ec13addcfa648cd94e31ba3830dd2a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:49.088495 14102 raft_consensus.cc:1261] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 1 FOLLOWER]: Refusing update from remote peer 7ec13addcfa648cd94e31ba3830dd2a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:49.088726 14329 consensus_queue.cc:1035] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dcd3b9f50e242e795fee06fe0dd78b4" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:49.088824 14323 consensus_queue.cc:1035] T cc329ac1e7804107a6aad1d5535202ab P 7ec13addcfa648cd94e31ba3830dd2a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15d50333f1c4dc8ad4f7030f6f46413" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:49.093276 14340 mvcc.cc:205] Tried to move back new op lower bound from 7299003142503890944 to 7299003142358298624. Current Snapshot: MvccSnapshot[applied={T|T < 7299003142503890944}]
I20260620 19:19:49.093981 13816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:49.094069 13816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:49.094828 13816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a87551286e2424bad949f835644504a: Sending DeleteTablet for 3 replicas of tablet cc329ac1e7804107a6aad1d5535202ab
I20260620 19:19:49.095134 14217 tablet_service.cc:1514] Processing DeleteTablet for tablet cc329ac1e7804107a6aad1d5535202ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:49 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:52412
I20260620 19:19:49.095147 13941 tablet_service.cc:1514] Processing DeleteTablet for tablet cc329ac1e7804107a6aad1d5535202ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:49 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:42766
I20260620 19:19:49.095566 14083 tablet_service.cc:1514] Processing DeleteTablet for tablet cc329ac1e7804107a6aad1d5535202ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:49 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50622
I20260620 19:19:49.095634 14341 mvcc.cc:205] Tried to move back new op lower bound from 7299003142503890944 to 7299003142358298624. Current Snapshot: MvccSnapshot[applied={T|T < 7299003142503890944}]
I20260620 19:19:49.096062 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13855
I20260620 19:19:49.099202 14350 tablet_replica.cc:330] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: stopping tablet replica
I20260620 19:19:49.099320 14350 raft_consensus.cc:2229] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:49.099423 14350 raft_consensus.cc:2258] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:49.099573 14347 tablet_replica.cc:330] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: stopping tablet replica
I20260620 19:19:49.099671 14347 raft_consensus.cc:2229] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:49.099761 14347 raft_consensus.cc:2258] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:49.100154 14347 ts_tablet_manager.cc:1905] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:49.101557 14347 ts_tablet_manager.cc:1918] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:49.101626 14347 log.cc:1196] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/cc329ac1e7804107a6aad1d5535202ab
I20260620 19:19:49.101894 14347 ts_tablet_manager.cc:1939] T cc329ac1e7804107a6aad1d5535202ab P 5dcd3b9f50e242e795fee06fe0dd78b4: Deleting consensus metadata
I20260620 19:19:49.102396 14350 ts_tablet_manager.cc:1905] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:19:49.103865 13801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:39607 (error 108)
I20260620 19:19:49.103905 14350 ts_tablet_manager.cc:1918] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:49.103962 14350 log.cc:1196] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/cc329ac1e7804107a6aad1d5535202ab
I20260620 19:19:49.104254 14350 ts_tablet_manager.cc:1939] T cc329ac1e7804107a6aad1d5535202ab P f15d50333f1c4dc8ad4f7030f6f46413: Deleting consensus metadata
I20260620 19:19:49.104643 13802 catalog_manager.cc:4841] TS f15d50333f1c4dc8ad4f7030f6f46413 (127.1.1.194:35883): tablet cc329ac1e7804107a6aad1d5535202ab (table test-table [id=b64e3abc094a4d0daac55971c818617b]) successfully deleted
I20260620 19:19:49.104707 14311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:49.104598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39607 (local address 127.0.0.1:42790)
0620 19:19:49.104643 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:49.104672 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:19:49.104810 13801 catalog_manager.cc:4584] TS 7ec13addcfa648cd94e31ba3830dd2a2 (127.1.1.193:39607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc329ac1e7804107a6aad1d5535202ab: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39607: connect: Connection refused (error 111)
I20260620 19:19:49.104982 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14019
I20260620 19:19:49.105118 13803 catalog_manager.cc:4841] TS 5dcd3b9f50e242e795fee06fe0dd78b4 (127.1.1.195:37711): tablet cc329ac1e7804107a6aad1d5535202ab (table test-table [id=b64e3abc094a4d0daac55971c818617b]) successfully deleted
I20260620 19:19:49.112134 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14152
I20260620 19:19:49.117744 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 13784
I20260620 19:19:49.123215 1031 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:19:49.467916 1031 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:19:49.468034 1031 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:19:49.811807 1031 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:19:49.811 UTC [12851] LOG: received smart shutdown request
2026-06-20 19:19:49.815 UTC [12851] LOG: background worker "logical replication launcher" (PID 12860) exited with exit code 1
2026-06-20 19:19:49.815 UTC [12855] LOG: shutting down
2026-06-20 19:19:49.956 UTC [12851] LOG: database system is shut down
2026-06-20T19:19:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43479 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:19:50 dist-test-slave-mwkb krb5kdc[14360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:19:50 dist-test-slave-mwkb krb5kdc[14360](info): set up 2 sockets
Jun 20 19:19:50 dist-test-slave-mwkb krb5kdc[14360](info): commencing operation
krb5kdc: starting...
W20260620 19:19:51.056737 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:19:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:19:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:19:51.163605 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44713
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:51.238479 14373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:51.238653 14373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:51.238675 14373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:51.239698 14373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:51.239729 14373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:51.239741 14373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:51.239753 14373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:51.239763 14373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:51.241163 14373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:51.241334 14373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:51.241484 14373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:51.245911 14381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:51.245903 14378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:51.246047 14373 server_base.cc:1034] running on GCE node
W20260620 19:19:51.246147 14379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:51.246393 14373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:51.246642 14373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:51.247799 14373 hybrid_clock.cc:648] HybridClock initialized: now 1781983191247782 us; error 46 us; skew 500 ppm
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:51.253937 14373 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:51.254376 14373 webserver.cc:460] Webserver started at http://127.1.1.254:43929/ using document root <none> and password file <none>
I20260620 19:19:51.254592 14373 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:51.254671 14373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:51.254820 14373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:51.255733 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "0c76a03205b547b0b98d86c0f2836d9e"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "bd87a3b8f96d5eb15a78a22b76b025f1"
server_key_iv: "4eb6e872c022a6586902ab39ffeeff36"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.256057 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "0c76a03205b547b0b98d86c0f2836d9e"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "bd87a3b8f96d5eb15a78a22b76b025f1"
server_key_iv: "4eb6e872c022a6586902ab39ffeeff36"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.257294 14373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:19:51.257928 14388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:51.258129 14373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:51.258224 14373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "0c76a03205b547b0b98d86c0f2836d9e"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "bd87a3b8f96d5eb15a78a22b76b025f1"
server_key_iv: "4eb6e872c022a6586902ab39ffeeff36"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.258306 14373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:51.273634 14373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:51.275219 14373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:51.275358 14373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:51.279065 14373 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44713
I20260620 19:19:51.279085 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44713 every 8 connection(s)
I20260620 19:19:51.279399 14373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:51.280081 14441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:51.281445 14441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:51.287938 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14373
I20260620 19:19:51.288041 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:19:51.288241 1031 external_mini_cluster.cc:1428] Setting key 97ad8992d347749b705288015c9a0fdb
I20260620 19:19:51.290555 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.282391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.1.1.254:44713)
0620 19:19:51.282504 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.282508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.282537 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:19:51.283020 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.283022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.283046 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.283154 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.283658 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.284424 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.285290 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.285578 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.285709 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.287383 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.287389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.287390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.287396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.289141 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.289495 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.289497 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.289498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.289618 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.289900 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.289901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.289901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.290073 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.290139 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.290339 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.290427 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":45,"threads_started":1}
I20260620 19:19:51.290707 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.282221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44713 (local address 127.0.0.1:44358)
0620 19:19:51.282378 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.282408 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.282854 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.283160 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.283162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.283414 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.283552 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.283556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.284533 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.284534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.285205 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.285207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.285679 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.285681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.285737 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.286103 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.286110 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.287293 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.289218 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.289224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.289226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.289404 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.289697 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.289698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.289698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.289832 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.290140 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.290146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.290217 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.290313 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.290530 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":62,"thread_start_us":84,"threads_started":1}
I20260620 19:19:51.291363 14441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:19:51.291666 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.291544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42662)
0620 19:19:51.291593 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.291636 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:19:51.291781 14389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
I20260620 19:19:51.292143 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.292066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42678)
0620 19:19:51.292104 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.292127 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:19:51.292253 14441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:35021: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.1.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
I20260620 19:19:51.310031 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:35021
--webserver_interface=127.1.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:51.341424 14441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } attempt: 1
I20260620 19:19:51.341840 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.341730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42682)
0620 19:19:51.341787 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.341820 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:19:51.342207 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.342130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42696)
0620 19:19:51.342170 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.342194 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:19:51.342322 14441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:35021: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
W20260620 19:19:51.383912 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:51.384089 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:51.384111 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:51.385205 14448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:51.385241 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:51.385254 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:51.385288 14448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:51.385314 14448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:51.386721 14448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:51.386874 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:51.387110 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:51.391582 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:51.391681 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:51.391624 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:51.391706 14448 server_base.cc:1034] running on GCE node
I20260620 19:19:51.391990 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:51.392283 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:51.393440 14448 hybrid_clock.cc:648] HybridClock initialized: now 1781983191393428 us; error 28 us; skew 500 ppm
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:51.395835 14448 init.cc:377] Logged in from keytab as kudu/127.1.1.253@KRBTEST.COM (short username kudu)
I20260620 19:19:51.396306 14448 webserver.cc:460] Webserver started at http://127.1.1.253:42253/ using document root <none> and password file <none>
I20260620 19:19:51.396525 14448 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:51.396592 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:51.396731 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:51.397771 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data/instance:
uuid: "2d16f480f239448289bd2b0da474cc72"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "cbf27d7ecd4e4c00ff08cbed39e3b70e"
server_key_iv: "403545dd18a45f73ec009934c31d845f"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.398131 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal/instance:
uuid: "2d16f480f239448289bd2b0da474cc72"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "cbf27d7ecd4e4c00ff08cbed39e3b70e"
server_key_iv: "403545dd18a45f73ec009934c31d845f"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.399374 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:19:51.400094 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:51.400261 14448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:51.400357 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "2d16f480f239448289bd2b0da474cc72"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "cbf27d7ecd4e4c00ff08cbed39e3b70e"
server_key_iv: "403545dd18a45f73ec009934c31d845f"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.400435 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:51.410470 14441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } attempt: 2
I20260620 19:19:51.410806 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.410696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42700)
0620 19:19:51.410750 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.410784 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:19:51.411100 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.411014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42714)
0620 19:19:51.411056 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.411078 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:19:51.411175 14441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:35021: Network error: Client connection negotiation failed: client connection to 127.1.1.253:35021: connect: Connection refused (error 111)
I20260620 19:19:51.419991 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:51.422125 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:51.422278 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:51.426116 14448 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.253:35021
I20260620 19:19:51.426156 14515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.253:35021 every 8 connection(s)
I20260620 19:19:51.426440 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
I20260620 19:19:51.427178 14516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:51.428462 14516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:51.433792 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14448
I20260620 19:19:51.433887 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-1/wal/instance
I20260620 19:19:51.434088 1031 external_mini_cluster.cc:1428] Setting key e1d85754e764662ad522e1c713c99d24
I20260620 19:19:51.436995 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.429008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44713 (local address 127.0.0.1:44366)
0620 19:19:51.429181 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.429210 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.429627 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.429852 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.429855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.430086 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.430261 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.430265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.430906 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.430907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.431643 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.431645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.431995 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.431996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.432103 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.432582 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.432589 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.434003 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.435829 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.435834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.435835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.436038 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.436237 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.436237 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.436237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.436322 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.436583 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.436586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.436682 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.436755 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.436854 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260620 19:19:51.437016 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.429028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.1.1.254:44713)
0620 19:19:51.429095 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.429096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.429283 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:19:51.429723 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.429724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.429736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.429793 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.430350 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.430808 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.431725 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.431908 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.431987 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.434151 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.434153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.434154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.434162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.435750 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.436111 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.436112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.436112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.436172 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.436384 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.436385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.436385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.436522 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.436587 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.436798 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.436896 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:19:51.437573 14516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.1.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.252@KRBTEST.COM" created.
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:19:51.446777 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.437860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.1.1.253:35021)
0620 19:19:51.438002 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.438006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.438020 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:19:51.438035 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.438036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.438044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.438127 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.438360 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.438991 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.439616 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.439882 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.439964 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.444043 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.444051 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.444052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.444059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.445609 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.445989 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.445991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.445992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.446038 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.446226 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.446226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.446226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.446335 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.446402 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.446580 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.446699 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260620 19:19:51.446782 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.437797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42716)
0620 19:19:51.437841 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.437850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.437882 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.438119 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.438120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.438252 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.438289 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.438291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.439068 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.439069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.439527 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.439528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.439963 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.439964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.440027 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.440403 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.440408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.443962 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.445688 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.445689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.445689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.445912 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.446100 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.446100 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.446101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.446168 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.446412 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.446413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.446457 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.446528 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.446637 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:19:51.447463 14516 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:19:51.447778 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.447653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60874)
0620 19:19:51.447700 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.447748 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:19:51.447918 14465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
I20260620 19:19:51.448369 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.448225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60886)
0620 19:19:51.448312 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.448342 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260620 19:19:51.448510 14516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:46713: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
I20260620 19:19:51.456852 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:46713
--webserver_interface=127.1.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:51.497709 14516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } attempt: 1
I20260620 19:19:51.498160 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.498023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60892)
0620 19:19:51.498098 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.498134 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 19:19:51.498469 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.498381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60904)
0620 19:19:51.498424 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.498446 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:19:51.498549 14516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:46713: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
I20260620 19:19:51.502301 14441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } attempt: 3
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:19:51.508590 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.502648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.1.1.253:35021)
0620 19:19:51.502698 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.502700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.502710 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:19:51.502757 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.502758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.502773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.502829 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.503132 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.503654 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.504362 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.504568 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.504644 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.506371 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.506372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.506372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.506378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.507560 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.507879 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.507880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.507881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.507930 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.508112 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.508112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.508113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.508210 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.508290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.508449 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.508515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:19:51.508602 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.502595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42722)
0620 19:19:51.502642 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.502651 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.502699 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.502846 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.502847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.502996 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.503055 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.503056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.503761 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.503763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.504264 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.504265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.504642 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.504643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.504725 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.505209 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.505214 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.506280 (+ 1066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.507647 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.507648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.507648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.507799 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.507993 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.507994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.507994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.508045 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.508283 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.508284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.508341 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.508415 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.508512 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:19:51.508937 14441 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:19:51.509261 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.509171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60918)
0620 19:19:51.509216 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.509240 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:19:51.509517 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.509444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60920)
0620 19:19:51.509482 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.509501 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:19:51.509603 14441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:46713: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
W20260620 19:19:51.534508 14523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:51.534727 14523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:51.534775 14523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:51.535821 14523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:51.535880 14523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:51.535903 14523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:51.535931 14523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:51.535951 14523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:51.537437 14523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44713,127.1.1.253:35021,127.1.1.252:46713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:46713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:51.537684 14523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:51.537883 14523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:19:51.540740 14441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } attempt: 1
I20260620 19:19:51.541122 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.540972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60924)
0620 19:19:51.541026 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.541094 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:19:51.541569 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.541442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60932)
0620 19:19:51.541507 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.541539 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:19:51.541746 14441 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:46713: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
W20260620 19:19:51.542335 14528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:51.542315 14529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:51.542315 14531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:51.542644 14523 server_base.cc:1034] running on GCE node
I20260620 19:19:51.542848 14523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:51.543111 14523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:51.544276 14523 hybrid_clock.cc:648] HybridClock initialized: now 1781983191544274 us; error 34 us; skew 500 ppm
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:51.546480 14523 init.cc:377] Logged in from keytab as kudu/127.1.1.252@KRBTEST.COM (short username kudu)
I20260620 19:19:51.546898 14523 webserver.cc:460] Webserver started at http://127.1.1.252:35923/ using document root <none> and password file <none>
I20260620 19:19:51.547118 14523 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:51.547183 14523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:51.547329 14523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:51.548276 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data/instance:
uuid: "30f974a79fba4c379233583b30727355"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "343d7371e3ad537b2596ea69751f70a0"
server_key_iv: "32753cbf10ac84346f558eedcf6aef92"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.548594 14523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal/instance:
uuid: "30f974a79fba4c379233583b30727355"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "343d7371e3ad537b2596ea69751f70a0"
server_key_iv: "32753cbf10ac84346f558eedcf6aef92"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.549865 14523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:19:51.550508 14538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:51.550668 14523 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:51.550760 14523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "30f974a79fba4c379233583b30727355"
format_stamp: "Formatted at 2026-06-20 19:19:51 on dist-test-slave-mwkb"
server_key: "343d7371e3ad537b2596ea69751f70a0"
server_key_iv: "32753cbf10ac84346f558eedcf6aef92"
server_key_version: "encryptionkey@0"
I20260620 19:19:51.550833 14523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:51.562595 14523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:51.564194 14523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:51.564340 14523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:51.566694 14516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } attempt: 2
I20260620 19:19:51.567063 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.566933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60940)
0620 19:19:51.567000 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.567029 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:19:51.567374 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.567297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60950)
0620 19:19:51.567327 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.567357 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 19:19:51.567462 14516 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:46713: Network error: Client connection negotiation failed: client connection to 127.1.1.252:46713: connect: Connection refused (error 111)
I20260620 19:19:51.568481 14523 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.252:46713
I20260620 19:19:51.568508 14590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.252:46713 every 8 connection(s)
I20260620 19:19:51.568818 14523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
I20260620 19:19:51.569640 14591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:51.570786 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14523
I20260620 19:19:51.570884 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781983021660275-1031-0/minicluster-data/master-2/wal/instance
I20260620 19:19:51.571070 1031 external_mini_cluster.cc:1428] Setting key 1e17595bc98779510fbcc0435f355a8a
I20260620 19:19:51.571240 14591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:19:51.580013 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.571942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.1.1.254:44713)
0620 19:19:51.571982 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.571985 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.572098 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:19:51.572428 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.572429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.572438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.572503 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.572962 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.573478 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.574549 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.574853 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.574927 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.577180 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.577182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.577182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.577190 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.578557 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.579030 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.579030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.579031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.579080 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.579351 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.579351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.579352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.579442 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.579510 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.579854 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.579920 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:19:51.580061 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.571857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44713 (local address 127.0.0.1:44378)
0620 19:19:51.571993 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.572031 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.572318 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.572532 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.572534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.572743 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.572880 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.572883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.573549 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.573550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.574475 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.574477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.574958 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.574959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.575091 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.575562 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.575569 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.577107 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.578648 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.578655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.578657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.578956 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.579149 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.579150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.579150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.579284 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.579505 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.579511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.579660 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.579807 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.579915 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":38,"thread_start_us":77,"threads_started":1}
I20260620 19:19:51.580593 14591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:19:51.581292 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.571532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.1.1.252:46713)
0620 19:19:51.571758 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.571764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.571783 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:19:51.572316 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.572324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.572346 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.572429 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.573037 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.574087 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.574954 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.575377 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.575426 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.577973 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.577980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.577981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.577988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.579723 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.580387 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.580389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.580389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.580445 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.580700 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.580701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.580701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.580832 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.580978 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.581101 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.581197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:51.587368 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.580832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:35021 (local address 127.0.0.1:42734)
0620 19:19:51.580885 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.580895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.580927 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.581041 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.581041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.581230 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.581265 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.581267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.582108 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.582108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.582701 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.582703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.583009 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.583010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.583069 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.583419 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.583422 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.584549 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.586448 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.586449 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.586449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.586672 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.586859 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.586859 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.586859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.586905 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.587149 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.587150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.587192 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.587256 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.587317 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:19:51.587468 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.580886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.1.1.253:35021)
0620 19:19:51.580923 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.580925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.580932 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:19:51.580978 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.580978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.580988 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.581039 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.581364 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.582019 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.582775 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.582945 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.582993 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.584641 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.584642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.584643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.584649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.586363 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.586739 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.586740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.586741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.586794 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.586962 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.586962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.586963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.587068 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.587144 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.587304 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.587377 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:19:51.587802 14591 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:19:51.588984 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.581978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.1.1.254:44713)
0620 19:19:51.582016 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.582018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.582026 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:51.582080 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.582081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.582090 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.582140 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.582555 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.583110 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.583909 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.584085 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.584136 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.586239 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.586240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.586240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.586247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.587868 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.588293 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.588294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.588295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.588344 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.588547 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.588547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.588548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.588649 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.588806 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.588838 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.588909 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:19:51.593966 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.588007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60960)
0620 19:19:51.588041 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.588048 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.588116 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.588220 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.588221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.588349 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.588371 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.588372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.588919 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.588920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.589578 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.589580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.589891 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.589891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.589930 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.590248 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.590252 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.591303 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.592744 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.592745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.592745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.592888 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.593126 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.593127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.593127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.593177 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.593666 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.593667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.593706 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.593781 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.593863 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:19:51.594064 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.588048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.1.1.252:46713)
0620 19:19:51.588115 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.588117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.588124 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:19:51.588160 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.588161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.588170 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.588217 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.588463 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.588842 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.589656 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.589834 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.589893 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.591385 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.591386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.591386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.591392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.592653 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.592964 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.592965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.592965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.593010 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.593259 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.593260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.593260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.593356 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.593430 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.593844 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.593926 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 19:19:51.595068 14591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: Bootstrap starting.
I20260620 19:19:51.595760 14591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:51.596015 14591 log.cc:824] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:51.596717 14591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: No bootstrap required, opened a new log
I20260620 19:19:51.597787 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.589494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.1.1.253:35021)
0620 19:19:51.589541 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.589542 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.589549 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:19:51.589603 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.589603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.589610 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.589656 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.590038 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.590420 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.591258 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.591439 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.591501 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.593423 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.593424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.593425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.593431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.596791 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.597120 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.597121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.597121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.597172 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.597342 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.597342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.597342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.597452 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.597612 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.597649 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.597714 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:19:51.598624 14591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.598744 14591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:51.598784 14591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f974a79fba4c379233583b30727355, State: Initialized, Role: FOLLOWER
I20260620 19:19:51.598886 14591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.599009 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:19:51.599166 14591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:51.599133 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.599248 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:19:51.601229 14612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:19:51.601281 14612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:19:51.601034 14523 master_runner.cc:186] Error getting master registration for 127.1.1.254:44713: OK, instance_id { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" instance_seqno: 1781983191277984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:19:51.603053 14523 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:19:51.608301 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.600759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.1.1.253:35021)
0620 19:19:51.600795 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.600797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.600803 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:19:51.600824 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.600825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.600831 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.600903 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.601507 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.602207 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.603144 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.603368 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.603397 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.605058 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.605060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.605060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.605094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.606635 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.606985 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.606986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.606986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.607037 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.607257 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.607258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.607258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.607379 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.607903 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.608141 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.608205 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:19:51.609230 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.600555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.1.1.254:44713)
0620 19:19:51.600729 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.600731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.601064 (+ 333us) server_negotiation.cc:407] Connection header received
0620 19:19:51.601176 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.601177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.601186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.601211 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.601911 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.602614 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.603610 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.603884 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.603958 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.605875 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.605876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.605876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.605882 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.607786 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.608144 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.608145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.608145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.608192 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.608370 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.608371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.608371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.608471 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.608529 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.609029 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.609148 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260620 19:19:51.610888 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.600603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.1.1.252:46713)
0620 19:19:51.600628 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.600630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.603744 (+ 3114us) server_negotiation.cc:407] Connection header received
0620 19:19:51.603833 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.603834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.603843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.603872 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.604589 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.605545 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.606535 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.606760 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.606817 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.608313 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.608314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.608315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.608321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.609899 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.610234 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.610234 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.610235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.610278 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.610459 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.610460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.610460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.610556 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.610624 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.610788 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.610839 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:19:51.616896 14441 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } attempt: 2
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:19:51.623126 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.617174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60990)
0620 19:19:51.617228 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.617237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.617277 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.617381 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.617381 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.617512 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.617538 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.617539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.618341 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.618342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.618822 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.618824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.619166 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.619167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.619230 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.619599 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.619604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.620798 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.622339 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.622340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.622340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.622487 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.622676 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.622677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.622677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.622722 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.622940 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.622940 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.622969 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.623015 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.623075 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:19:51.623190 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.617236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.1.1.252:46713)
0620 19:19:51.617278 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.617280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.617287 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:19:51.617326 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.617327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.617335 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.617380 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.617611 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.618109 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.618905 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.619091 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.619167 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.620877 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.620878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.620878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.620885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.622256 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.622551 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.622552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.622553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.622605 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.622775 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.622776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.622776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.622884 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.622943 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.623047 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.623111 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:19:51.624667 14441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: Bootstrap starting.
I20260620 19:19:51.625367 14441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:51.625684 14441 log.cc:824] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:51.626385 14441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: No bootstrap required, opened a new log
I20260620 19:19:51.628333 14441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.628460 14441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:51.628492 14441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c76a03205b547b0b98d86c0f2836d9e, State: Initialized, Role: FOLLOWER
I20260620 19:19:51.628604 14441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.628857 14618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.628890 14441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:51.628958 14618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:19:51.630618 14629 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:19:51.630714 14629 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:19:51.631186 14373 master_runner.cc:186] Error getting master registration for 127.1.1.253:35021: OK, instance_id { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" instance_seqno: 1781983191425005 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:19:51.633857 14373 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:19:51.658613 14516 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } attempt: 3
Jun 20 19:19:51 dist-test-slave-mwkb krb5kdc[14360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983191, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:19:51.666669 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.658878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:46713 (local address 127.0.0.1:60998)
0620 19:19:51.658925 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:51.658936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:51.658973 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:51.659126 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:51.659127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:51.659309 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:51.659352 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.659354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.660226 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.660227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.661036 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:51.661038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.661816 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.661817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:51.661879 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.662328 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:51.662333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:51.663769 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:51.665573 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.665574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.665575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.665804 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.666009 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:51.666010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:51.666010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:51.666075 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:51.666368 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:51.666368 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:51.666411 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:19:51.666484 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:19:51.666571 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:19:51.666674 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:51.658932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.1.1.252:46713)
0620 19:19:51.658968 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:51.658970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:51.658978 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:51.659025 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:51.659025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:51.659034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:51.659103 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:51.659424 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.660143 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.661466 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:51.661745 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:51.661809 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:51.663858 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:51.663860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:51.663860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:51.663867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:51.665483 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.665881 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.665882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.665883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.665937 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:51.666141 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:51.666142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:51.666142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:51.666284 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:51.666362 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:51.666519 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 19:19:51.666588 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:19:51.668380 14516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: Bootstrap starting.
I20260620 19:19:51.669133 14516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:51.669414 14516 log.cc:824] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:51.670284 14516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: No bootstrap required, opened a new log
I20260620 19:19:51.672185 14516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.672302 14516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:51.672336 14516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d16f480f239448289bd2b0da474cc72, State: Initialized, Role: FOLLOWER
I20260620 19:19:51.672549 14516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.672941 14516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:51.672920 14632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.673108 14632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:19:51.677722 14643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:19:51.679451 14643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:19:51.681190 14448 master_runner.cc:263] No leader master found from master 2d16f480f239448289bd2b0da474cc72
I20260620 19:19:51.681241 14448 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:19:51.776039 14394 tablet.cc:2369] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:19:51.814899 14544 tablet.cc:2369] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:19:51.922973 14469 tablet.cc:2369] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:19:51.985179 14632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:51.985282 14632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.985505 14632 leader_election.cc:288] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c76a03205b547b0b98d86c0f2836d9e (127.1.1.254:44713), 30f974a79fba4c379233583b30727355 (127.1.1.252:46713)
I20260620 19:19:51.985759 14565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d16f480f239448289bd2b0da474cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f974a79fba4c379233583b30727355" is_pre_election: true
I20260620 19:19:51.985786 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d16f480f239448289bd2b0da474cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c76a03205b547b0b98d86c0f2836d9e" is_pre_election: true
I20260620 19:19:51.985899 14565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d16f480f239448289bd2b0da474cc72 in term 0.
I20260620 19:19:51.985962 14415 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d16f480f239448289bd2b0da474cc72 in term 0.
I20260620 19:19:51.986110 14465 leader_election.cc:302] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d16f480f239448289bd2b0da474cc72, 30f974a79fba4c379233583b30727355; no voters:
I20260620 19:19:51.986231 14632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:19:51.986292 14632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:19:51.986343 14632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:51.987026 14632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.987157 14632 leader_election.cc:288] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [CANDIDATE]: Term 1 election: Requested vote from peers 0c76a03205b547b0b98d86c0f2836d9e (127.1.1.254:44713), 30f974a79fba4c379233583b30727355 (127.1.1.252:46713)
I20260620 19:19:51.987268 14415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d16f480f239448289bd2b0da474cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c76a03205b547b0b98d86c0f2836d9e"
I20260620 19:19:51.987294 14565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d16f480f239448289bd2b0da474cc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30f974a79fba4c379233583b30727355"
I20260620 19:19:51.987360 14565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:51.987357 14415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:51.988178 14565 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d16f480f239448289bd2b0da474cc72 in term 1.
I20260620 19:19:51.988178 14415 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d16f480f239448289bd2b0da474cc72 in term 1.
I20260620 19:19:51.988349 14467 leader_election.cc:302] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c76a03205b547b0b98d86c0f2836d9e, 2d16f480f239448289bd2b0da474cc72; no voters:
I20260620 19:19:51.988461 14632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:51.988620 14632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [term 1 LEADER]: Becoming Leader. State: Replica: 2d16f480f239448289bd2b0da474cc72, State: Running, Role: LEADER
I20260620 19:19:51.988724 14632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } }
I20260620 19:19:51.989127 14645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d16f480f239448289bd2b0da474cc72. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.989212 14645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:51.989385 14647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:51.989563 14647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:51.990849 14565 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [term 1 FOLLOWER]: Refusing update from remote peer 2d16f480f239448289bd2b0da474cc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:51.990864 14415 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [term 1 FOLLOWER]: Refusing update from remote peer 2d16f480f239448289bd2b0da474cc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:51.991118 14632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:51.991282 14645 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:51.992373 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d16f480f239448289bd2b0da474cc72. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.992472 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:19:51.992588 14618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d16f480f239448289bd2b0da474cc72. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.992667 14618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:19:51.993216 14645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.993302 14645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:51.993306 14597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.993371 14597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:19:51.993396 14632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.993463 14632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:51.993884 14618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d16f480f239448289bd2b0da474cc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c76a03205b547b0b98d86c0f2836d9e" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44713 } } peers { permanent_uuid: "2d16f480f239448289bd2b0da474cc72" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 35021 } } peers { permanent_uuid: "30f974a79fba4c379233583b30727355" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 46713 } } }
I20260620 19:19:51.993966 14618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:19:51.994138 14647 catalog_manager.cc:1347] Generated new cluster ID: 0b585d3d35e841b18de606d3eb690dc3
I20260620 19:19:51.994179 14647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:52.033306 14647 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:19:52.033957 14647 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:52.038566 14647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d16f480f239448289bd2b0da474cc72: Generated new TSK 0
I20260620 19:19:52.038746 14647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:19:52.601641 14612 catalog_manager.cc:1259] Loaded cluster ID: 0b585d3d35e841b18de606d3eb690dc3
I20260620 19:19:52.601702 14612 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: loading cluster ID for follower catalog manager: success
I20260620 19:19:52.602434 14612 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: acquiring CA information for follower catalog manager: success
I20260620 19:19:52.602676 14612 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 30f974a79fba4c379233583b30727355: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:19:52.631222 14629 catalog_manager.cc:1259] Loaded cluster ID: 0b585d3d35e841b18de606d3eb690dc3
I20260620 19:19:52.631287 14629 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: loading cluster ID for follower catalog manager: success
I20260620 19:19:52.632458 14629 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: acquiring CA information for follower catalog manager: success
I20260620 19:19:52.632818 14629 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c76a03205b547b0b98d86c0f2836d9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:19:53.094373 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14373
I20260620 19:19:53.099306 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14448
I20260620 19:19:53.103945 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14523
2026-06-20T19:19:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3042 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:19:53 dist-test-slave-mwkb krb5kdc[14662](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:19:53 dist-test-slave-mwkb krb5kdc[14662](info): set up 2 sockets
Jun 20 19:19:53 dist-test-slave-mwkb krb5kdc[14662](info): commencing operation
krb5kdc: starting...
W20260620 19:19:54.080869 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:19:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:19:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:19:54.181707 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38409
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:38409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:54.253224 14675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:54.253386 14675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:54.253405 14675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:54.254482 14675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:54.254521 14675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:54.254536 14675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:54.254549 14675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:54.254598 14675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:54.256021 14675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:38409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:54.256253 14675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:54.256472 14675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:54.260665 14680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:54.260689 14683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:54.260717 14681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:54.260802 14675 server_base.cc:1034] running on GCE node
I20260620 19:19:54.261173 14675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:54.261463 14675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:54.262614 14675 hybrid_clock.cc:648] HybridClock initialized: now 1781983194262604 us; error 25 us; skew 500 ppm
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14662](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983194, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:54.264837 14675 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:54.265290 14675 webserver.cc:460] Webserver started at http://127.1.1.254:36105/ using document root <none> and password file <none>
I20260620 19:19:54.265479 14675 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:54.265537 14675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:54.265669 14675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:54.266522 14675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "b19dcb8d7819406ba248f19d6adfaa02"
format_stamp: "Formatted at 2026-06-20 19:19:54 on dist-test-slave-mwkb"
server_key: "f7b191f18d2d0db82940090a5ff76374"
server_key_iv: "70b01e3caf7238221cfaa02beb4efc94"
server_key_version: "encryptionkey@0"
I20260620 19:19:54.266834 14675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "b19dcb8d7819406ba248f19d6adfaa02"
format_stamp: "Formatted at 2026-06-20 19:19:54 on dist-test-slave-mwkb"
server_key: "f7b191f18d2d0db82940090a5ff76374"
server_key_iv: "70b01e3caf7238221cfaa02beb4efc94"
server_key_version: "encryptionkey@0"
I20260620 19:19:54.268007 14675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:19:54.268669 14690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:54.268827 14675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:54.268918 14675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "b19dcb8d7819406ba248f19d6adfaa02"
format_stamp: "Formatted at 2026-06-20 19:19:54 on dist-test-slave-mwkb"
server_key: "f7b191f18d2d0db82940090a5ff76374"
server_key_iv: "70b01e3caf7238221cfaa02beb4efc94"
server_key_version: "encryptionkey@0"
I20260620 19:19:54.268993 14675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:54.276539 14675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:54.278549 14675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:54.278703 14675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:54.282612 14675 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:38409
I20260620 19:19:54.282647 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:38409 every 8 connection(s)
I20260620 19:19:54.282931 14675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:54.283614 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:54.284993 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14675
I20260620 19:19:54.285135 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:19:54.285311 1031 external_mini_cluster.cc:1428] Setting key dd9bbbdba7072792036a232075dd495e
I20260620 19:19:54.285952 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02: Bootstrap starting.
I20260620 19:19:54.286489 14743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:54.286782 14743 log.cc:824] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:54.287465 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02: No bootstrap required, opened a new log
I20260620 19:19:54.289443 14743 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } }
I20260620 19:19:54.289574 14743 raft_consensus.cc:374] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:54.289630 14743 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19dcb8d7819406ba248f19d6adfaa02, State: Initialized, Role: FOLLOWER
I20260620 19:19:54.289747 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } }
I20260620 19:19:54.289830 14743 raft_consensus.cc:388] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:54.289871 14743 raft_consensus.cc:482] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:54.289907 14743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:54.290354 14743 raft_consensus.cc:504] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } }
I20260620 19:19:54.290462 14743 leader_election.cc:302] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b19dcb8d7819406ba248f19d6adfaa02; no voters:
I20260620 19:19:54.290625 14743 leader_election.cc:288] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:19:54.290663 14748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:54.290804 14748 raft_consensus.cc:686] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [term 1 LEADER]: Becoming Leader. State: Replica: b19dcb8d7819406ba248f19d6adfaa02, State: Running, Role: LEADER
I20260620 19:19:54.290838 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:54.290957 14748 consensus_queue.cc:237] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } }
I20260620 19:19:54.291320 14750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b19dcb8d7819406ba248f19d6adfaa02. Latest consensus state: current_term: 1 leader_uuid: "b19dcb8d7819406ba248f19d6adfaa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } } }
I20260620 19:19:54.291384 14750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14662](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:54.291574 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b19dcb8d7819406ba248f19d6adfaa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19dcb8d7819406ba248f19d6adfaa02" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38409 } } }
I20260620 19:19:54.291878 14754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:54.291891 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02 [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:54.292418 14754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:54.294001 14754 catalog_manager.cc:1347] Generated new cluster ID: 30c88945ba6640859bd4f68e3934d03e
I20260620 19:19:54.294051 14754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:54.295208 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:54.285733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.1.1.254:38409)
0620 19:19:54.285883 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:54.285891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:54.285918 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:19:54.286292 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:54.286298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:54.286337 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:54.286435 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:54.287211 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:54.288130 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:54.289206 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:54.289540 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:54.289646 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:54.291983 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:54.291987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:54.291988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:54.291998 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:54.293694 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:54.294067 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:54.294070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:54.294070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:54.294174 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:54.294392 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:54.294393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:54.294393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:54.294653 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:54.294785 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:54.294907 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:19:54.295022 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260620 19:19:54.295998 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:19:54.312654 14754 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:19:54.313287 14754 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:54.321928 14754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b19dcb8d7819406ba248f19d6adfaa02: Generated new TSK 0
I20260620 19:19:54.322098 14754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:19:54.349570 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14675
2026-06-20T19:19:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1244 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14770](info): set up 2 sockets
Jun 20 19:19:54 dist-test-slave-mwkb krb5kdc[14770](info): commencing operation
krb5kdc: starting...
W20260620 19:19:55.313001 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:19:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:19:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:19:55.415060 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:55.487941 14783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:55.488245 14783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:55.488283 14783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:55.489488 14783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:55.489540 14783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:55.489562 14783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:55.489585 14783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:55.489606 14783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:55.490912 14783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:55.491154 14783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:55.491346 14783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:55.495471 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.495481 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.495479 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:55.495698 14783 server_base.cc:1034] running on GCE node
I20260620 19:19:55.495970 14783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:55.496209 14783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:55.497356 14783 hybrid_clock.cc:648] HybridClock initialized: now 1781983195497344 us; error 26 us; skew 500 ppm
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:55.499536 14783 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:55.499866 14783 webserver.cc:460] Webserver started at http://127.1.1.254:42593/ using document root <none> and password file <none>
I20260620 19:19:55.500038 14783 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:55.500074 14783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:55.500149 14783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:55.501001 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "2e6e59444ae6482fa8f84572ddfa7bed"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "af4c12dce1e638dd55e486d5c4893a46"
server_key_iv: "53d3a150d9498014b554e9299c7a0fc7"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.501318 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "2e6e59444ae6482fa8f84572ddfa7bed"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "af4c12dce1e638dd55e486d5c4893a46"
server_key_iv: "53d3a150d9498014b554e9299c7a0fc7"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.502382 14783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:55.503012 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.503166 14783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:55.503271 14783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2e6e59444ae6482fa8f84572ddfa7bed"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "af4c12dce1e638dd55e486d5c4893a46"
server_key_iv: "53d3a150d9498014b554e9299c7a0fc7"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.503353 14783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:55.510222 14783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:55.512074 14783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:55.512164 14783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:55.515712 14783 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44641
I20260620 19:19:55.515738 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44641 every 8 connection(s)
I20260620 19:19:55.516010 14783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:55.516842 14851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:55.518530 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14783
I20260620 19:19:55.518605 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:19:55.518769 1031 external_mini_cluster.cc:1428] Setting key 856638f6cbcc12f77fceacffeea3106c
I20260620 19:19:55.518949 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap starting.
I20260620 19:19:55.519627 14851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:55.519929 14851 log.cc:824] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:55.520551 14851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: No bootstrap required, opened a new log
I20260620 19:19:55.522125 14851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:55.522249 14851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:55.522269 14851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Initialized, Role: FOLLOWER
I20260620 19:19:55.522383 14851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:55.522446 14851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:55.522465 14851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:55.522511 14851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:55.523036 14851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:55.523146 14851 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6e59444ae6482fa8f84572ddfa7bed; no voters:
I20260620 19:19:55.523356 14851 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:19:55.523394 14856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:55.523613 14856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 LEADER]: Becoming Leader. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Running, Role: LEADER
I20260620 19:19:55.523618 14851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:55.523727 14856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:55.524072 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6e59444ae6482fa8f84572ddfa7bed. Latest consensus state: current_term: 1 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:55.524209 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:55.524466 14865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:55.524971 14865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:55.525535 14857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:55.525642 14857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:55.526875 14865 catalog_manager.cc:1347] Generated new cluster ID: a7f48a000cb74f1797bcea93a24a43ac
I20260620 19:19:55.526927 14865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:55.531548 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.519242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.1.1.254:44641)
0620 19:19:55.519481 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:55.519489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:55.519517 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:19:55.520078 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:55.520087 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:55.520143 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:55.520310 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:55.520923 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.521737 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.522543 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.522961 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.523087 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.525816 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:55.525822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:55.525822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:55.525835 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:55.530004 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.530461 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.530463 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.530463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.530548 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.530772 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.530773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.530773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.531025 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:55.531159 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:55.531282 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:19:55.531389 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260620 19:19:55.544226 14865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:19:55.544704 14865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:55.549423 14865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Generated new TSK 0
I20260620 19:19:55.549587 14865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:19:55.574081 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:55.667943 14877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:55.668118 14877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:55.668140 14877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:55.669302 14877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:55.669339 14877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:55.669364 14877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:55.670776 14877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:55.671003 14877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:55.671221 14877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:55.675832 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.675830 14882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.675803 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:55.676045 14877 server_base.cc:1034] running on GCE node
I20260620 19:19:55.676222 14877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:55.676466 14877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:55.677624 14877 hybrid_clock.cc:648] HybridClock initialized: now 1781983195677613 us; error 48 us; skew 500 ppm
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:55.679926 14877 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:55.680335 14877 webserver.cc:460] Webserver started at http://127.1.1.193:37879/ using document root <none> and password file <none>
I20260620 19:19:55.680526 14877 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:55.680585 14877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:55.680716 14877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:55.681639 14877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "18c51fb6346ba7e32854f5d054e15969"
server_key_iv: "cf8fd89693ff7c4be81be89702283092"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.681948 14877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "18c51fb6346ba7e32854f5d054e15969"
server_key_iv: "cf8fd89693ff7c4be81be89702283092"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.683063 14877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:19:55.683682 14892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.683840 14877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:55.683925 14877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "18c51fb6346ba7e32854f5d054e15969"
server_key_iv: "cf8fd89693ff7c4be81be89702283092"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.684017 14877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:55.700254 14877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:55.702175 14877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:55.702312 14877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:55.702507 14877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:55.702767 14877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:55.702813 14877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.702840 14877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:55.702862 14877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.708271 14877 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:42077
I20260620 19:19:55.708334 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:42077 every 8 connection(s)
I20260620 19:19:55.708596 14877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:55.716459 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.709142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:55951 (local address 127.1.1.254:44641)
0620 19:19:55.709206 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:55.709208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:55.709447 (+ 239us) server_negotiation.cc:407] Connection header received
0620 19:19:55.709858 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:55.709859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:55.709870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:55.709918 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:55.710740 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.711162 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.712052 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.712259 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.712327 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.714092 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:55.714094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:55.714094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:55.714101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:55.715243 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.715589 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.715590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.715591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.715646 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.715900 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.715901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.715901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.716015 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:55.716079 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:55.716305 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 19:19:55.716380 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:19:55.716588 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.709159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:55951)
0620 19:19:55.709308 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:55.709361 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:55.709764 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:55.709972 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:55.709975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:55.710498 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:55.710641 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.710644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.711259 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.711260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.711965 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.711967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.712345 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.712346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.712458 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.712790 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:55.712797 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:55.713973 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:55.715328 (+ 1355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:55.715334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:55.715336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:55.715522 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:55.715715 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:55.715716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:55.715716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:55.715818 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:55.716072 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:55.716078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:55.716189 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 19:19:55.716276 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:19:55.716397 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260620 19:19:55.716964 15006 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:55.717036 15006 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:55.717192 15006 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:55.717571 14815 ts_manager.cc:194] Registered new tserver with Master: 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077)
I20260620 19:19:55.718060 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:55951
I20260620 19:19:55.718372 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 14877
I20260620 19:19:55.718479 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:19:55.718641 1031 external_mini_cluster.cc:1428] Setting key 32ef359c1e418dc9027edffa7ecb7343
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:19:55.740793 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:55.814011 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:55.814194 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:55.814216 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:55.815338 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:55.815376 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:55.815429 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:55.816779 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:55.816992 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:55.817215 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:55.822091 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.822098 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.822103 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:55.822418 15011 server_base.cc:1034] running on GCE node
I20260620 19:19:55.822667 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:55.822872 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:55.824047 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781983195824020 us; error 22 us; skew 500 ppm
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:55.826604 15011 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:55.826953 15011 webserver.cc:460] Webserver started at http://127.1.1.194:39419/ using document root <none> and password file <none>
I20260620 19:19:55.827131 15011 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:55.827169 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:55.827250 15011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:55.828179 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "40208add5a19dfbd282ba115152ced0c"
server_key_iv: "f31f6024776575ac2fd519a4b8fb1c5c"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.828516 15011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "40208add5a19dfbd282ba115152ced0c"
server_key_iv: "f31f6024776575ac2fd519a4b8fb1c5c"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.830212 15011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:19:55.830894 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.831161 15011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:19:55.831238 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "40208add5a19dfbd282ba115152ced0c"
server_key_iv: "f31f6024776575ac2fd519a4b8fb1c5c"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.831290 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:55.851907 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:55.853391 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:55.853484 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:55.853637 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:55.853837 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:55.853863 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.853879 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:55.853891 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.859181 15011 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:46617
I20260620 19:19:55.859220 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:46617 every 8 connection(s)
I20260620 19:19:55.859516 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:55.864615 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15011
I20260620 19:19:55.864740 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:19:55.864912 1031 external_mini_cluster.cc:1428] Setting key 6a0aa0f77033f59702018b3f3f06c726
I20260620 19:19:55.867833 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.860197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:39665 (local address 127.1.1.254:44641)
0620 19:19:55.860256 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:55.860258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:55.860446 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:19:55.860840 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:55.860841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:55.860849 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:55.860898 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:55.861659 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.862092 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.862948 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.863138 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.863190 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.865184 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:55.865185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:55.865185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:55.865191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:55.866579 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.866995 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.866996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.866996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.867046 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:55.867244 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:55.867245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:55.867245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:55.867366 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:55.867438 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:55.867680 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:19:55.867753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:19:55.867982 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.860155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:39665)
0620 19:19:55.860315 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:55.860350 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:55.860769 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:55.860939 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:55.860942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:55.861419 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:55.861571 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.861574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.862181 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.862182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.862857 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.862858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.863214 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.863215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.863342 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.863718 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:55.863725 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:55.865032 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:55.866649 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:55.866654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:55.866655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:55.866906 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:55.867110 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:55.867110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:55.867110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:55.867186 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:55.867431 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:55.867437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:55.867549 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 19:19:55.867658 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:19:55.867789 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260620 19:19:55.868490 15140 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:55.868578 15140 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:55.868729 15140 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:55.869088 14815 ts_manager.cc:194] Registered new tserver with Master: 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
I20260620 19:19:55.869405 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:39665
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:19:55.887156 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:55.959843 15145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:55.960029 15145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:55.960049 15145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:55.961153 15145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:55.961194 15145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:55.961242 15145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:55.962584 15145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:55.962792 15145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:55.962988 15145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:55.968116 15151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.968108 15153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:55.968152 15150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:55.968165 15145 server_base.cc:1034] running on GCE node
I20260620 19:19:55.968502 15145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:55.968726 15145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:55.969858 15145 hybrid_clock.cc:648] HybridClock initialized: now 1781983195969846 us; error 27 us; skew 500 ppm
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:55.972000 15145 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:55.972357 15145 webserver.cc:460] Webserver started at http://127.1.1.195:44505/ using document root <none> and password file <none>
I20260620 19:19:55.972556 15145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:55.972617 15145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:55.972752 15145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:19:55.973666 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "00f842ed75294c849eb034964c91dd9b"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "569eb6ee3ec0438f885f0ff349b020ff"
server_key_iv: "437359b99422bad73f50e016c64c2f98"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.973982 15145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "00f842ed75294c849eb034964c91dd9b"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "569eb6ee3ec0438f885f0ff349b020ff"
server_key_iv: "437359b99422bad73f50e016c64c2f98"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.975234 15145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:19:55.975855 15160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.975996 15145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:55.976079 15145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "00f842ed75294c849eb034964c91dd9b"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "569eb6ee3ec0438f885f0ff349b020ff"
server_key_iv: "437359b99422bad73f50e016c64c2f98"
server_key_version: "encryptionkey@0"
I20260620 19:19:55.976152 15145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:55.985551 15145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:55.987555 15145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:55.987697 15145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:55.987884 15145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:55.988161 15145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:55.988212 15145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.988258 15145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:55.988286 15145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:55.993573 15145 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:45125
I20260620 19:19:55.993608 15273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:45125 every 8 connection(s)
I20260620 19:19:55.993878 15145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:19:55 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983195, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:56.001227 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15145
I20260620 19:19:56.001348 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:19:56.001549 1031 external_mini_cluster.cc:1428] Setting key 7cb49cc414ea69a5a27525d9639a0ad5
I20260620 19:19:56.002199 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.994539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:39961 (local address 127.1.1.254:44641)
0620 19:19:55.994614 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:55.994616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:55.994794 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:19:55.995170 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:55.995171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:55.995186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:55.995233 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:55.996343 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.996825 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.997706 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.997862 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.997909 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.999788 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:55.999789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:55.999789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:55.999795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.000761 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.001108 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.001112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.001112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.001154 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.001464 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.001465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.001465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.001574 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.001668 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.002009 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.002104 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:19:56.002331 15276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:55.994489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:39961)
0620 19:19:55.994676 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:55.994717 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:55.995095 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:55.995288 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:55.995291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:55.995990 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:55.996267 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.996271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.996896 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.996897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.997638 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:55.997639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:55.997925 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:55.997926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:55.998001 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:55.998484 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:55.998491 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:55.999712 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:56.000821 (+ 1109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.000826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.000827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.000995 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.001214 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.001215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.001215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.001376 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.001659 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:56.001665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:56.001856 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.001964 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.002108 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
I20260620 19:19:56.002730 15274 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:56.002820 15274 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:56.002969 15274 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:56.003304 14815 ts_manager.cc:194] Registered new tserver with Master: 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125)
I20260620 19:19:56.003574 14815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:39961
I20260620 19:19:56.013378 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:19:56.019378 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.014576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58276 (local address 127.1.1.254:44641)
0620 19:19:56.014628 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.014629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.014716 (+ 87us) server_negotiation.cc:407] Connection header received
0620 19:19:56.014762 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.014763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.014770 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.014814 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:56.015310 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.015689 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.016317 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.016457 (+ 140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.016497 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.017473 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:56.017474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:56.017474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:56.017479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.018445 (+ 966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.018715 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.018716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.018716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.018758 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.018911 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.018911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.018912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.019065 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.019143 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.019258 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.019311 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:19:56.020578 14815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:19:56.022097 14815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:19:56.028301 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45125 (local address 127.0.0.1:36088)
0620 19:19:56.024172 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.024212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.024274 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.024625 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.024628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.024631 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.024691 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.024693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.026038 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.026039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.027175 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.027177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.028008 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.028009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.028084 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028097 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.028162 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.028214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":63,"thread_start_us":99,"threads_started":1}
I20260620 19:19:56.028430 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42077 (local address 127.0.0.1:39100)
0620 19:19:56.024523 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.024533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.024588 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.024884 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.024885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.024886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.024925 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.024926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.026230 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.026230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.027435 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.027438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.028178 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.028179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.028243 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028248 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.028300 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.028341 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":30,"spinlock_wait_cycles":768,"thread_start_us":61,"threads_started":1}
I20260620 19:19:56.028596 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.1.1.193:42077)
0620 19:19:56.024629 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.024633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.024642 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:19:56.024679 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.024680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.024719 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.024800 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.025005 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.026129 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.027553 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.028081 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.028156 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028209 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.028474 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.028518 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260620 19:19:56.028610 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.1.1.195:45125)
0620 19:19:56.024443 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.024448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.024458 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:19:56.024492 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.024494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.024510 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.024766 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.024822 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.025940 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.027283 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.027917 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.027990 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028040 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.028509 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.028544 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260620 19:19:56.028993 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:46617 (local address 127.0.0.1:44246)
0620 19:19:56.024522 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.024533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.024588 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.024904 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.024905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.024907 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.024958 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.024960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.026153 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.026154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.027462 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.027463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.028306 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.028307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.028803 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028807 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.028856 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.028926 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":36,"thread_start_us":61,"threads_started":1}
I20260620 19:19:56.029026 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.024353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.1.1.194:46617)
0620 19:19:56.024540 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.024544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.024664 (+ 120us) server_negotiation.cc:407] Connection header received
0620 19:19:56.024731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.024734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.024749 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.024823 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.025127 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.026048 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.027551 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.028235 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.028307 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.028369 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.028906 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.028949 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260620 19:19:56.029333 15208 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de (DEFAULT_TABLE table=test-table [id=a13c1681500e4381852db6011af61fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.029646 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2d18c3d07456cb8d83205ee5d21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.029742 14940 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de (DEFAULT_TABLE table=test-table [id=a13c1681500e4381852db6011af61fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.030020 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2d18c3d07456cb8d83205ee5d21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.029978 15074 tablet_service.cc:1467] Processing CreateTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de (DEFAULT_TABLE table=test-table [id=a13c1681500e4381852db6011af61fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.030289 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b2d18c3d07456cb8d83205ee5d21de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.032624 15288 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Bootstrap starting.
I20260620 19:19:56.032940 15289 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Bootstrap starting.
I20260620 19:19:56.032979 15290 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap starting.
I20260620 19:19:56.033442 15288 tablet_bootstrap.cc:654] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.033593 15289 tablet_bootstrap.cc:654] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.033663 15290 tablet_bootstrap.cc:654] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.034052 15288 log.cc:824] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.034173 15290 log.cc:824] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.034219 15289 log.cc:824] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.034708 15288 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: No bootstrap required, opened a new log
I20260620 19:19:56.034786 15288 ts_tablet_manager.cc:1397] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:56.035006 15289 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: No bootstrap required, opened a new log
I20260620 19:19:56.035080 15289 ts_tablet_manager.cc:1397] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:19:56.035238 15290 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: No bootstrap required, opened a new log
I20260620 19:19:56.035295 15290 ts_tablet_manager.cc:1397] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:56.036350 15288 raft_consensus.cc:348] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.036495 15288 raft_consensus.cc:374] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.036535 15288 raft_consensus.cc:729] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb98ca2a1348ccb5a4aaaf2c018936, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.036629 15288 consensus_queue.cc:260] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.036873 15288 ts_tablet_manager.cc:1428] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:56.036813 15289 raft_consensus.cc:348] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.036950 15289 raft_consensus.cc:374] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.036882 15140 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.036993 15289 raft_consensus.cc:729] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f842ed75294c849eb034964c91dd9b, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.036999 15290 raft_consensus.cc:348] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.037153 15290 raft_consensus.cc:374] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.037127 15289 consensus_queue.cc:260] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.037185 15290 raft_consensus.cc:729] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e659cdeb06a48bbb8e92ffff29bc9c9, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.037279 15290 consensus_queue.cc:260] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.037391 15289 ts_tablet_manager.cc:1428] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 19:19:56.037495 15274 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.037503 15290 ts_tablet_manager.cc:1428] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:56.037604 15006 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.097970 15296 raft_consensus.cc:482] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:56.098047 15296 raft_consensus.cc:504] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.098440 15296 leader_election.cc:288] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
I20260620 19:19:56.101234 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.098644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:41291 (local address 127.1.1.194:46617)
0620 19:19:56.098681 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.098684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.098814 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:19:56.098885 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.098886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.098898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.098965 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.099099 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.099516 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100592 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.100920 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100994 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.101002 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.101147 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.101176 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:19:56.101243 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.098640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:33475 (local address 127.1.1.195:45125)
0620 19:19:56.098682 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.098684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.098721 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:19:56.098782 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.098783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.098793 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.098859 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.099005 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.099444 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100536 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.100898 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100975 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.100984 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.101156 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.101187 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:19:56.101253 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.098575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45125 (local address 127.1.1.193:33475)
0620 19:19:56.098665 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.098677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.098733 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.098873 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.098874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.098875 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.098936 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.098937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.099524 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.099525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.100443 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.100445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.100975 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.101051 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.101054 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.101143 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.101183 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 19:19:56.101426 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.098578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:46617 (local address 127.1.1.193:41291)
0620 19:19:56.098744 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.098752 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.098826 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.098967 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.098967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.098969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.099035 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.099037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.099596 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.099597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.100502 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.100504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.100995 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.100995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.101061 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.101088 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.101141 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.101175 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":10,"thread_start_us":88,"threads_started":1}
I20260620 19:19:56.101701 15228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2d18c3d07456cb8d83205ee5d21de" candidate_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f842ed75294c849eb034964c91dd9b" is_pre_election: true
I20260620 19:19:56.101732 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2d18c3d07456cb8d83205ee5d21de" candidate_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" is_pre_election: true
I20260620 19:19:56.101832 15094 raft_consensus.cc:2454] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e659cdeb06a48bbb8e92ffff29bc9c9 in term 0.
I20260620 19:19:56.101832 15228 raft_consensus.cc:2454] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e659cdeb06a48bbb8e92ffff29bc9c9 in term 0.
I20260620 19:19:56.102039 14895 leader_election.cc:302] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f842ed75294c849eb034964c91dd9b, 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters:
I20260620 19:19:56.102172 15296 raft_consensus.cc:2781] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:19:56.102233 15296 raft_consensus.cc:482] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:19:56.102264 15296 raft_consensus.cc:3037] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.102806 15296 raft_consensus.cc:504] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.102921 15296 leader_election.cc:288] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 1 election: Requested vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
I20260620 19:19:56.103076 15228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2d18c3d07456cb8d83205ee5d21de" candidate_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f842ed75294c849eb034964c91dd9b"
I20260620 19:19:56.103075 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b2d18c3d07456cb8d83205ee5d21de" candidate_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
I20260620 19:19:56.103154 15228 raft_consensus.cc:3037] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.103154 15094 raft_consensus.cc:3037] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.103880 15228 raft_consensus.cc:2454] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e659cdeb06a48bbb8e92ffff29bc9c9 in term 1.
I20260620 19:19:56.103880 15094 raft_consensus.cc:2454] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e659cdeb06a48bbb8e92ffff29bc9c9 in term 1.
I20260620 19:19:56.104033 14895 leader_election.cc:302] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f842ed75294c849eb034964c91dd9b, 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters:
I20260620 19:19:56.104182 15296 raft_consensus.cc:2781] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:56.104344 15296 raft_consensus.cc:686] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e659cdeb06a48bbb8e92ffff29bc9c9, State: Running, Role: LEADER
I20260620 19:19:56.104449 15296 consensus_queue.cc:237] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.105166 14813 catalog_manager.cc:5495] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } health_report { overall_health: HEALTHY } } }
W20260620 19:19:56.110023 15141 tablet.cc:2369] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:19:56.122079 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.120154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.1.1.193:42077)
0620 19:19:56.120189 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.120190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.120196 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:19:56.120274 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.120275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.120288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.120338 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:19:56.120431 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.120794 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.121477 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.121628 (+ 151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.121673 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.121809 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:19:56.121882 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:19:56.121935 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.121967 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.121993 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:19:56.123402 15228 raft_consensus.cc:1261] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Refusing update from remote peer 4e659cdeb06a48bbb8e92ffff29bc9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:56.123411 15094 raft_consensus.cc:1261] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Refusing update from remote peer 4e659cdeb06a48bbb8e92ffff29bc9c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:56.123803 15298 consensus_queue.cc:1035] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:56.123924 15296 consensus_queue.cc:1035] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:56.126448 15303 mvcc.cc:205] Tried to move back new op lower bound from 7299003171319259136 to 7299003171244371968. Current Snapshot: MvccSnapshot[applied={T|T < 7299003171319259136}]
I20260620 19:19:56.126768 15302 mvcc.cc:205] Tried to move back new op lower bound from 7299003171319259136 to 7299003171244371968. Current Snapshot: MvccSnapshot[applied={T|T < 7299003171319259136}]
I20260620 19:19:56.128345 14815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:56.128434 14815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:56.129139 14815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Sending DeleteTablet for 3 replicas of tablet f1b2d18c3d07456cb8d83205ee5d21de
I20260620 19:19:56.129482 15208 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:36088
I20260620 19:19:56.129602 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14877
I20260620 19:19:56.129619 14940 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39100
I20260620 19:19:56.129765 15074 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44246
I20260620 19:19:56.129935 15311 tablet_replica.cc:330] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: stopping tablet replica
I20260620 19:19:56.130012 15311 raft_consensus.cc:2229] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:56.130112 15311 raft_consensus.cc:2258] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.130234 15310 tablet_replica.cc:330] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: stopping tablet replica
I20260620 19:19:56.130319 15310 raft_consensus.cc:2229] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:56.130409 15310 raft_consensus.cc:2258] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.130509 15311 ts_tablet_manager.cc:1905] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:56.130756 15310 ts_tablet_manager.cc:1905] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:56.132275 15311 ts_tablet_manager.cc:1918] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:56.132285 15310 ts_tablet_manager.cc:1918] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:56.132331 15310 log.cc:1196] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/f1b2d18c3d07456cb8d83205ee5d21de
I20260620 19:19:56.132335 15311 log.cc:1196] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/f1b2d18c3d07456cb8d83205ee5d21de
I20260620 19:19:56.132520 15310 ts_tablet_manager.cc:1939] T f1b2d18c3d07456cb8d83205ee5d21de P 00f842ed75294c849eb034964c91dd9b: Deleting consensus metadata
I20260620 19:19:56.132593 15311 ts_tablet_manager.cc:1939] T f1b2d18c3d07456cb8d83205ee5d21de P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting consensus metadata
I20260620 19:19:56.132835 14801 catalog_manager.cc:4841] TS 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): tablet f1b2d18c3d07456cb8d83205ee5d21de (table test-table [id=a13c1681500e4381852db6011af61fe8]) successfully deleted
I20260620 19:19:56.133185 14800 catalog_manager.cc:4841] TS 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): tablet f1b2d18c3d07456cb8d83205ee5d21de (table test-table [id=a13c1681500e4381852db6011af61fe8]) successfully deleted
W20260620 19:19:56.136236 14799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:42077 (error 108)
I20260620 19:19:56.136392 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15011
I20260620 19:19:56.136582 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.136494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42077 (local address 127.0.0.1:39126)
0620 19:19:56.136528 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.136552 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:19:56.136665 14799 catalog_manager.cc:4584] TS 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1b2d18c3d07456cb8d83205ee5d21de: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42077: connect: Connection refused (error 111)
I20260620 19:19:56.140846 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15145
I20260620 19:19:56.144948 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 14783
I20260620 19:19:56.149459 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--webserver_interface=127.1.1.254
--webserver_port=42593
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:56.223088 15313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.223290 15313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.223326 15313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.224484 15313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:56.224552 15313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.224575 15313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.224598 15313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:56.224627 15313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:56.226065 15313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.226337 15313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.226552 15313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.234709 15322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.234706 15320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.234704 15319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.234958 15313 server_base.cc:1034] running on GCE node
I20260620 19:19:56.235242 15313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.235575 15313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.236732 15313 hybrid_clock.cc:648] HybridClock initialized: now 1781983196236729 us; error 33 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.239086 15313 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:56.239490 15313 webserver.cc:460] Webserver started at http://127.1.1.254:42593/ using document root <none> and password file <none>
I20260620 19:19:56.239718 15313 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.239801 15313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.241055 15313 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:19:56.241739 15329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.241901 15313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:19:56.241997 15313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2e6e59444ae6482fa8f84572ddfa7bed"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "af4c12dce1e638dd55e486d5c4893a46"
server_key_iv: "53d3a150d9498014b554e9299c7a0fc7"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.242362 15313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.265823 15313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.267550 15313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.267695 15313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.271467 15313 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44641
I20260620 19:19:56.271493 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44641 every 8 connection(s)
I20260620 19:19:56.271822 15313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:56.273218 15382 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:19:56.273564 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15313
I20260620 19:19:56.273845 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:42077
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=37879
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:56.274121 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap starting.
I20260620 19:19:56.282526 15382 log.cc:824] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.284344 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:19:56.284608 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap complete.
I20260620 19:19:56.286734 15382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.286947 15382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.287125 15382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.287211 15382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:56.287263 15382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:56.287307 15382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:19:56.288158 15382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.288264 15382 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6e59444ae6482fa8f84572ddfa7bed; no voters:
I20260620 19:19:56.288405 15382 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:19:56.288462 15386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:19:56.288617 15386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 LEADER]: Becoming Leader. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Running, Role: LEADER
I20260620 19:19:56.288650 15382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:56.288713 15386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.288964 15386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6e59444ae6482fa8f84572ddfa7bed. Latest consensus state: current_term: 2 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:56.289108 15386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:56.288964 15387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:56.289186 15387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:56.289695 15390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:56.290409 15390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:56.290515 15390 catalog_manager.cc:1259] Loaded cluster ID: a7f48a000cb74f1797bcea93a24a43ac
I20260620 19:19:56.290838 15390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:19:56.291112 15390 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:19:56.295644 15390 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:19:56.297801 15390 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:19:56.297966 15390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Loaded TSK: 0
I20260620 19:19:56.298174 15390 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:19:56.358636 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.358805 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.358836 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.359992 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.360029 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.360054 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:56.361416 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37879
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.361601 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.361800 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.365933 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.366041 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.365943 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.366268 15384 server_base.cc:1034] running on GCE node
I20260620 19:19:56.366456 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.366725 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.367904 15384 hybrid_clock.cc:648] HybridClock initialized: now 1781983196367886 us; error 37 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.370433 15384 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:56.370826 15384 webserver.cc:460] Webserver started at http://127.1.1.193:37879/ using document root <none> and password file <none>
I20260620 19:19:56.371007 15384 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.371075 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.372239 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:19:56.372879 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.373034 15384 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:19:56.373137 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "18c51fb6346ba7e32854f5d054e15969"
server_key_iv: "cf8fd89693ff7c4be81be89702283092"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.373420 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.401489 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.403434 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.403574 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.403771 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:56.404232 15426 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:19:56.405635 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:19:56.405679 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:19:56.405710 15384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:19:56.406278 15384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:19:56.406324 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:19:56.406354 15426 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap starting.
I20260620 19:19:56.413432 15384 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:42077
I20260620 19:19:56.413491 15533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:42077 every 8 connection(s)
I20260620 19:19:56.413897 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:19:56.416411 15426 log.cc:824] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.418203 15426 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:19:56.418468 15426 tablet_bootstrap.cc:492] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap complete.
I20260620 19:19:56.419368 15426 ts_tablet_manager.cc:1397] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260620 19:19:56.419919 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15384
I20260620 19:19:56.420178 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:46617
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=39419
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:56.420706 15426 raft_consensus.cc:348] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.421036 15426 raft_consensus.cc:729] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e659cdeb06a48bbb8e92ffff29bc9c9, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.421201 15426 consensus_queue.cc:260] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.421445 15426 ts_tablet_manager.cc:1428] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:56.425410 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.414538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:55103)
0620 19:19:56.414722 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.414760 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.415244 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.416938 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.416941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.417249 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:56.417408 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.417412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.418483 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.418484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.419301 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.419303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.420020 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.420021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.420147 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.420623 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:56.420631 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:56.422085 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:56.423814 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.423820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.423822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.424060 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.424334 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.424334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.424335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.424439 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.424798 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:56.424804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:56.424935 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.425041 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.425222 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260620 19:19:56.425410 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.414637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:55103 (local address 127.1.1.254:44641)
0620 19:19:56.416627 (+ 1990us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.416634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.416656 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:19:56.416748 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.416752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.416787 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.416912 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:56.417519 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.418382 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.419409 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.419882 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.419988 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.422182 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:56.422186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:56.422186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:56.422196 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.423704 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.424158 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.424161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.424161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.424251 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.424513 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.424514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.424514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.424720 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.424806 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.425114 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.425240 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1951,"thread_start_us":100,"threads_started":1}
I20260620 19:19:56.426098 15534 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:56.426168 15534 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:56.426349 15534 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:56.427035 15346 ts_manager.cc:194] Registered new tserver with Master: 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077)
I20260620 19:19:56.428314 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:55103
I20260620 19:19:56.429133 15534 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.432127 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.427685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.1.1.193:42077)
0620 19:19:56.428854 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.428859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.428868 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:19:56.428904 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.428906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.429007 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.429104 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.429276 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.430043 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.431432 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.431835 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.431885 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.431913 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.432065 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.432088 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":62,"threads_started":1}
I20260620 19:19:56.432161 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.427640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42077 (local address 127.0.0.1:39136)
0620 19:19:56.428614 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.428648 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.428724 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.429128 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.429130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.429133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.429189 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.429191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.430118 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.430119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.431347 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.431350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.431901 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.431902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.431976 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.431987 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.432050 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.432095 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":88,"threads_started":1}
I20260620 19:19:56.432587 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b2d18c3d07456cb8d83205ee5d21de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39136
I20260620 19:19:56.432999 15543 tablet_replica.cc:330] stopping tablet replica
I20260620 19:19:56.433107 15543 raft_consensus.cc:2229] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:56.433167 15543 raft_consensus.cc:2258] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.433409 15543 ts_tablet_manager.cc:1905] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:56.435495 15543 ts_tablet_manager.cc:1918] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:56.435572 15543 log.cc:1196] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/f1b2d18c3d07456cb8d83205ee5d21de
I20260620 19:19:56.435798 15543 ts_tablet_manager.cc:1939] T f1b2d18c3d07456cb8d83205ee5d21de P 4e659cdeb06a48bbb8e92ffff29bc9c9: Deleting consensus metadata
I20260620 19:19:56.436228 15330 catalog_manager.cc:4841] TS 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077): tablet f1b2d18c3d07456cb8d83205ee5d21de (table test-table [id=a13c1681500e4381852db6011af61fe8]) successfully deleted
W20260620 19:19:56.496515 15539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.496671 15539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.496691 15539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.497653 15539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.497684 15539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.497706 15539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:56.498971 15539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:46617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=39419
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.499121 15539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.499272 15539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.503679 15548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.503695 15549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.503870 15539 server_base.cc:1034] running on GCE node
W20260620 19:19:56.503959 15551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.504233 15539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.504498 15539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.505641 15539 hybrid_clock.cc:648] HybridClock initialized: now 1781983196505622 us; error 39 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.507825 15539 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:56.508211 15539 webserver.cc:460] Webserver started at http://127.1.1.194:39419/ using document root <none> and password file <none>
I20260620 19:19:56.508416 15539 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.508495 15539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.509680 15539 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:19:56.510185 15558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.510337 15539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:56.510421 15539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "40208add5a19dfbd282ba115152ced0c"
server_key_iv: "f31f6024776575ac2fd519a4b8fb1c5c"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.510689 15539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.522462 15539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.524503 15539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.524652 15539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.524858 15539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:56.525194 15539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:56.525246 15539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.525286 15539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:56.525314 15539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.530615 15539 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:46617
I20260620 19:19:56.530714 15671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:46617 every 8 connection(s)
I20260620 19:19:56.530993 15539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:19:56.534933 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15539
I20260620 19:19:56.535240 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:45125
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=44505
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:56.539554 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.531691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:33841 (local address 127.1.1.254:44641)
0620 19:19:56.531767 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.531770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.531927 (+ 157us) server_negotiation.cc:407] Connection header received
0620 19:19:56.532322 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.532323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.532336 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.532378 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:56.533173 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.533615 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.534486 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.534676 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.534725 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.536806 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:56.536808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:56.536809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:56.536822 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.538073 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.538575 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.538576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.538576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.538624 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.538923 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.538924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.538924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.539044 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.539095 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.539438 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.539498 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:19:56.539805 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.531652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:33841)
0620 19:19:56.531821 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.531852 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.532235 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.532426 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.532429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.532935 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:56.533092 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.533120 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.533685 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.533685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.534415 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.534417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.534765 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.534766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.534910 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.535364 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:56.535374 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:56.536730 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:56.538180 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.538186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.538188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.538501 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.538693 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.538694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.538694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.538832 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.539114 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:56.539120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:56.539290 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.539421 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.539577 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260620 19:19:56.540354 15672 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:56.540452 15672 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:56.540639 15672 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:56.540985 15346 ts_manager.cc:194] Registered new tserver with Master: 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
I20260620 19:19:56.541353 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:33841
W20260620 19:19:56.611327 15675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.611522 15675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.611559 15675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.612615 15675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.612653 15675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.612676 15675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:56.614215 15675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:45125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=44505
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.614379 15675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.614585 15675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.618999 15680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.619014 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.619010 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.619061 15675 server_base.cc:1034] running on GCE node
I20260620 19:19:56.619405 15675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.619632 15675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.620781 15675 hybrid_clock.cc:648] HybridClock initialized: now 1781983196620775 us; error 34 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.622929 15675 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:56.623294 15675 webserver.cc:460] Webserver started at http://127.1.1.195:44505/ using document root <none> and password file <none>
I20260620 19:19:56.623504 15675 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.623574 15675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.625280 15675 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:56.625849 15690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.626000 15675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:56.626106 15675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "00f842ed75294c849eb034964c91dd9b"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "569eb6ee3ec0438f885f0ff349b020ff"
server_key_iv: "437359b99422bad73f50e016c64c2f98"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.626399 15675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.640786 15675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.642392 15675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.642529 15675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.642740 15675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:56.643039 15675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:56.643088 15675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.643123 15675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:56.643146 15675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.648401 15675 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:45125
I20260620 19:19:56.648471 15803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:45125 every 8 connection(s)
I20260620 19:19:56.648734 15675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:19:56.650279 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15675
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:19:56.656680 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.650726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.1.1.254:44641)
0620 19:19:56.650829 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.650831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.650841 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:19:56.650896 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.650896 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.650907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.650982 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:56.651593 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.652179 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.652830 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.653006 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.653071 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.654191 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:56.654193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:56.654193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:56.654200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.655620 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.655917 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.655918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.655918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.655970 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.656151 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.656151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.656152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.656251 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.656415 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.656447 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.656612 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":54,"threads_started":1}
I20260620 19:19:56.658063 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.649361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:47055 (local address 127.1.1.254:44641)
0620 19:19:56.649421 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.649423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.649597 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:19:56.649994 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.649995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.650003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.650054 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:56.651152 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.651829 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.652786 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.653008 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.653107 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.655377 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:56.655378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:56.655378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:56.655384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:56.656730 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.657235 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.657236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.657236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.657284 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:56.657511 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:56.657512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:56.657512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:56.657610 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:56.657678 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.657922 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.657989 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":20}
I20260620 19:19:56.658258 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.649327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:47055)
0620 19:19:56.649483 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.649519 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.649918 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.650098 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.650100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.650814 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:56.651059 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.651065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.651905 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.651906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.652710 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.652712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.653109 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.653109 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.653213 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.653724 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:56.653731 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:56.655275 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:56.656824 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.656831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.656833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.657145 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.657347 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:56.657348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:56.657348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:56.657447 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:56.657670 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:56.657676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:56.657807 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.657903 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.658030 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260620 19:19:56.658658 15804 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44641
I20260620 19:19:56.658725 15804 heartbeater.cc:464] Registering TS with master...
I20260620 19:19:56.658835 15804 heartbeater.cc:511] Master 127.1.1.254:44641 requested a full tablet report, sending...
I20260620 19:19:56.659109 15346 ts_manager.cc:194] Registered new tserver with Master: 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125)
I20260620 19:19:56.659459 15346 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:47055
I20260620 19:19:56.668064 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:19:56.670542 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.668511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.1.1.254:44641)
0620 19:19:56.668560 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.668562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.668617 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:19:56.668681 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.668682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.668712 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.668769 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:19:56.668860 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.669339 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.669989 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.670151 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.670198 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.670320 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:19:56.670393 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:19:56.670446 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.670480 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.670504 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:19:56.670769 15346 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:19:56.672197 15346 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:19:56.674212 15468 tablet_service.cc:1467] Processing CreateTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca (DEFAULT_TABLE table=test-table [id=d25ed74838654d859f29fd0c4597ef79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.674401 15468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b750af5918483f9e84bb0ebe5d85ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.675967 15426 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap starting.
I20260620 19:19:56.676225 15426 tablet_bootstrap.cc:654] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.676712 15426 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: No bootstrap required, opened a new log
I20260620 19:19:56.676774 15426 ts_tablet_manager.cc:1397] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:19:56.676869 15426 raft_consensus.cc:348] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.676937 15426 raft_consensus.cc:374] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.676977 15426 raft_consensus.cc:729] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e659cdeb06a48bbb8e92ffff29bc9c9, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.677035 15426 consensus_queue.cc:260] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.677178 15426 ts_tablet_manager.cc:1428] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:56.677865 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.673906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45125 (local address 127.0.0.1:36102)
0620 19:19:56.673937 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.673947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.673984 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.674530 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.674531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.674533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.674568 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.674570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.675627 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.675628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.676672 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.676673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.677649 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.677650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.677712 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.677715 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.677766 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.677806 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 19:19:56.677878 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.674224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.1.1.195:45125)
0620 19:19:56.674366 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.674371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.674379 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:56.674410 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.674412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.674425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.674636 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.674688 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.675536 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.676770 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.677522 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.677605 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.677657 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.677782 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.677813 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260620 19:19:56.678038 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.673983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:46617 (local address 127.0.0.1:44254)
0620 19:19:56.674109 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.674119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.674182 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.674576 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.674577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.674579 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.674610 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.674611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.675742 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.675743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.676761 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.676762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.677778 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.677779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.677823 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.677826 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.677903 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.677951 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260620 19:19:56.678496 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.674241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.1.1.194:46617)
0620 19:19:56.674428 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.674433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.674444 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:19:56.674483 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.674485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.674499 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.674582 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.674657 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.675644 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.676849 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.677626 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.677689 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.677742 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.678375 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.678420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260620 19:19:56.678580 15738 tablet_service.cc:1467] Processing CreateTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca (DEFAULT_TABLE table=test-table [id=d25ed74838654d859f29fd0c4597ef79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.678887 15738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b750af5918483f9e84bb0ebe5d85ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.679093 15606 tablet_service.cc:1467] Processing CreateTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca (DEFAULT_TABLE table=test-table [id=d25ed74838654d859f29fd0c4597ef79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:19:56.679402 15606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6b750af5918483f9e84bb0ebe5d85ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:19:56.681243 15814 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Bootstrap starting.
I20260620 19:19:56.681811 15815 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Bootstrap starting.
I20260620 19:19:56.681891 15814 tablet_bootstrap.cc:654] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.682349 15815 tablet_bootstrap.cc:654] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Neither blocks nor log segments found. Creating new log.
I20260620 19:19:56.682387 15814 log.cc:824] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.682854 15815 log.cc:824] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.683079 15814 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: No bootstrap required, opened a new log
I20260620 19:19:56.683146 15814 ts_tablet_manager.cc:1397] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:19:56.683471 15815 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: No bootstrap required, opened a new log
I20260620 19:19:56.683544 15815 ts_tablet_manager.cc:1397] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:19:56.684407 15814 raft_consensus.cc:348] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.684530 15814 raft_consensus.cc:374] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.684551 15814 raft_consensus.cc:729] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f842ed75294c849eb034964c91dd9b, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.684664 15814 consensus_queue.cc:260] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.684932 15814 ts_tablet_manager.cc:1428] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:19:56.684962 15804 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.684904 15815 raft_consensus.cc:348] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.685098 15815 raft_consensus.cc:374] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:19:56.685143 15815 raft_consensus.cc:729] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bcb98ca2a1348ccb5a4aaaf2c018936, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.685256 15815 consensus_queue.cc:260] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.685542 15815 ts_tablet_manager.cc:1428] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:19:56.685554 15672 heartbeater.cc:503] Master 127.1.1.254:44641 was elected leader, sending a full tablet report...
I20260620 19:19:56.709614 15819 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:56.709673 15819 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.709877 15819 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077)
I20260620 19:19:56.712301 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.710017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42077 (local address 127.1.1.194:47253)
0620 19:19:56.710052 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.710062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.710103 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.710246 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.710246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.710259 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.710320 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.710322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.710886 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.710887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.711508 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.711510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.712085 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.712086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.712151 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.712155 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.712230 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.712263 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 19:19:56.712338 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.710093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:47253 (local address 127.1.1.193:42077)
0620 19:19:56.710135 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.710137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.710145 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:56.710169 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.710170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.710180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.710230 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.710381 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.710804 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.711585 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.712009 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.712059 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.712066 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.712236 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.712269 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:19:56.712424 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.710043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:56965 (local address 127.1.1.195:45125)
0620 19:19:56.710087 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.710089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.710196 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:19:56.710278 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.710279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.710288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.710333 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:19:56.710462 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.710803 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.711630 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.712115 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.712174 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.712181 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.712354 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.712383 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:19:56.712448 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.710017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45125 (local address 127.1.1.194:56965)
0620 19:19:56.710145 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.710155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.710207 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:56.710339 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:56.710340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:56.710341 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:19:56.710410 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.710411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.710889 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.710890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.711529 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:56.711530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.712215 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.712216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:56.712284 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.712288 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:19:56.712336 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:19:56.712374 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":11,"thread_start_us":68,"threads_started":1}
I20260620 19:19:56.712631 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b750af5918483f9e84bb0ebe5d85ca" candidate_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" is_pre_election: true
I20260620 19:19:56.712764 15488 raft_consensus.cc:2454] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcb98ca2a1348ccb5a4aaaf2c018936 in term 0.
I20260620 19:19:56.712937 15559 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bcb98ca2a1348ccb5a4aaaf2c018936, 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters:
I20260620 19:19:56.712965 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b750af5918483f9e84bb0ebe5d85ca" candidate_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f842ed75294c849eb034964c91dd9b" is_pre_election: true
I20260620 19:19:56.713057 15758 raft_consensus.cc:2454] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bcb98ca2a1348ccb5a4aaaf2c018936 in term 0.
I20260620 19:19:56.713115 15819 raft_consensus.cc:2781] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:19:56.713163 15819 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:19:56.713193 15819 raft_consensus.cc:3037] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.713857 15819 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.713981 15819 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [CANDIDATE]: Term 1 election: Requested vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077)
I20260620 19:19:56.714123 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b750af5918483f9e84bb0ebe5d85ca" candidate_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00f842ed75294c849eb034964c91dd9b"
I20260620 19:19:56.714170 15488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6b750af5918483f9e84bb0ebe5d85ca" candidate_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
I20260620 19:19:56.714221 15758 raft_consensus.cc:3037] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.714232 15488 raft_consensus.cc:3037] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:19:56.714911 15758 raft_consensus.cc:2454] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcb98ca2a1348ccb5a4aaaf2c018936 in term 1.
I20260620 19:19:56.715059 15561 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f842ed75294c849eb034964c91dd9b, 0bcb98ca2a1348ccb5a4aaaf2c018936; no voters:
I20260620 19:19:56.715157 15819 raft_consensus.cc:2781] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:19:56.715298 15819 raft_consensus.cc:686] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 LEADER]: Becoming Leader. State: Replica: 0bcb98ca2a1348ccb5a4aaaf2c018936, State: Running, Role: LEADER
I20260620 19:19:56.715284 15488 raft_consensus.cc:2454] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bcb98ca2a1348ccb5a4aaaf2c018936 in term 1.
I20260620 19:19:56.715355 15819 consensus_queue.cc:237] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:56.716029 15346 catalog_manager.cc:5495] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194). New cstate: current_term: 1 leader_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } health_report { overall_health: UNKNOWN } } }
I20260620 19:19:56.722637 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:56.720410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.1.1.194:46617)
0620 19:19:56.720477 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:56.720479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:56.720487 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:19:56.720545 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:56.720546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:56.720560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:56.720665 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:19:56.720796 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.721274 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.721951 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:56.722130 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:56.722200 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:56.722338 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:19:56.722425 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:19:56.722500 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:56.722544 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:19:56.722560 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:19:56.724298 15758 raft_consensus.cc:1261] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Refusing update from remote peer 0bcb98ca2a1348ccb5a4aaaf2c018936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:56.724383 15488 raft_consensus.cc:1261] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Refusing update from remote peer 0bcb98ca2a1348ccb5a4aaaf2c018936: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:19:56.724673 15821 consensus_queue.cc:1035] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:56.724803 15819 consensus_queue.cc:1035] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:19:56.726557 15824 mvcc.cc:205] Tried to move back new op lower bound from 7299003173779869696 to 7299003173746688000. Current Snapshot: MvccSnapshot[applied={T|T < 7299003173779869696}]
I20260620 19:19:56.726670 15825 mvcc.cc:205] Tried to move back new op lower bound from 7299003173779869696 to 7299003173746688000. Current Snapshot: MvccSnapshot[applied={T|T < 7299003173779869696}]
I20260620 19:19:56.727878 15826 mvcc.cc:205] Tried to move back new op lower bound from 7299003173779869696 to 7299003173746688000. Current Snapshot: MvccSnapshot[applied={T|T < 7299003173779869696 or (T in {7299003173779869696})}]
I20260620 19:19:56.728880 15345 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:56.728968 15345 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:19:56.729723 15345 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Sending DeleteTablet for 3 replicas of tablet f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:19:56.729991 15738 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:36102
I20260620 19:19:56.730050 15468 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39136
I20260620 19:19:56.730129 15543 tablet_replica.cc:330] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: stopping tablet replica
I20260620 19:19:56.730149 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15384
I20260620 19:19:56.730157 15606 tablet_service.cc:1514] Processing DeleteTablet for tablet f6b750af5918483f9e84bb0ebe5d85ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:19:56 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:44254
I20260620 19:19:56.730171 15543 raft_consensus.cc:2229] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:56.730234 15543 raft_consensus.cc:2258] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.730468 15835 tablet_replica.cc:330] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: stopping tablet replica
I20260620 19:19:56.730578 15835 raft_consensus.cc:2229] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:19:56.730695 15835 raft_consensus.cc:2258] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.731114 15835 ts_tablet_manager.cc:1905] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:56.732880 15835 ts_tablet_manager.cc:1918] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:56.732995 15835 log.cc:1196] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:19:56.733197 15834 tablet_replica.cc:330] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: stopping tablet replica
I20260620 19:19:56.733294 15834 raft_consensus.cc:2229] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:19:56.733390 15834 raft_consensus.cc:2258] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:19:56.733523 15835 ts_tablet_manager.cc:1939] T f6b750af5918483f9e84bb0ebe5d85ca P 0bcb98ca2a1348ccb5a4aaaf2c018936: Deleting consensus metadata
I20260620 19:19:56.733783 15834 ts_tablet_manager.cc:1905] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:19:56.733961 15331 catalog_manager.cc:4841] TS 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): tablet f6b750af5918483f9e84bb0ebe5d85ca (table test-table [id=d25ed74838654d859f29fd0c4597ef79]) successfully deleted
I20260620 19:19:56.735601 15834 ts_tablet_manager.cc:1918] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:19:56.735667 15834 log.cc:1196] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:19:56.735910 15834 ts_tablet_manager.cc:1939] T f6b750af5918483f9e84bb0ebe5d85ca P 00f842ed75294c849eb034964c91dd9b: Deleting consensus metadata
I20260620 19:19:56.736269 15332 catalog_manager.cc:4841] TS 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): tablet f6b750af5918483f9e84bb0ebe5d85ca (table test-table [id=d25ed74838654d859f29fd0c4597ef79]) successfully deleted
W20260620 19:19:56.736487 15330 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:42077 (error 108)
W20260620 19:19:56.736919 15810 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:19:56.736813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:42077 (local address 127.0.0.1:39138)
0620 19:19:56.736849 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.736881 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42077: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 19:19:56.737136 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15539
W20260620 19:19:56.737015 15330 catalog_manager.cc:4584] TS 4e659cdeb06a48bbb8e92ffff29bc9c9 (127.1.1.193:42077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6b750af5918483f9e84bb0ebe5d85ca: Network error: Client connection negotiation failed: client connection to 127.1.1.193:42077: connect: Connection reset by peer (error 104)
I20260620 19:19:56.742089 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15675
I20260620 19:19:56.746415 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15313
I20260620 19:19:56.751328 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--webserver_interface=127.1.1.254
--webserver_port=42593
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:56.822350 15837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.822512 15837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.822531 15837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.823570 15837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:19:56.823611 15837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.823649 15837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.823676 15837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:19:56.823693 15837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:19:56.825186 15837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=42593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.825416 15837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.825651 15837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.830309 15845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.830363 15843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.830351 15842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.830392 15837 server_base.cc:1034] running on GCE node
I20260620 19:19:56.830694 15837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.830960 15837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.832115 15837 hybrid_clock.cc:648] HybridClock initialized: now 1781983196832112 us; error 33 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.834285 15837 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:19:56.834682 15837 webserver.cc:460] Webserver started at http://127.1.1.254:42593/ using document root <none> and password file <none>
I20260620 19:19:56.834890 15837 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.834956 15837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.836108 15837 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:56.836623 15852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.836772 15837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:56.836856 15837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2e6e59444ae6482fa8f84572ddfa7bed"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "af4c12dce1e638dd55e486d5c4893a46"
server_key_iv: "53d3a150d9498014b554e9299c7a0fc7"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.837167 15837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.853866 15837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.855469 15837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.855605 15837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.859161 15837 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44641
I20260620 19:19:56.859207 15904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44641 every 8 connection(s)
I20260620 19:19:56.859534 15837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:19:56.860855 15905 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:19:56.861436 15905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap starting.
I20260620 19:19:56.865537 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15837
I20260620 19:19:56.865793 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:42077
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=37879
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:56.869472 15905 log.cc:824] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:56.871258 15905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:19:56.871500 15905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed: Bootstrap complete.
I20260620 19:19:56.873461 15905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.873641 15905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Initialized, Role: FOLLOWER
I20260620 19:19:56.873790 15905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.873842 15905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:19:56.873896 15905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:19:56.873935 15905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:19:56.874830 15905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.874908 15905 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6e59444ae6482fa8f84572ddfa7bed; no voters:
I20260620 19:19:56.875077 15905 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:19:56.875108 15909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:19:56.875252 15909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [term 3 LEADER]: Becoming Leader. State: Replica: 2e6e59444ae6482fa8f84572ddfa7bed, State: Running, Role: LEADER
I20260620 19:19:56.875342 15905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:19:56.875399 15909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } }
I20260620 19:19:56.875700 15910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:56.875770 15910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:56.875959 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6e59444ae6482fa8f84572ddfa7bed. Latest consensus state: current_term: 3 leader_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6e59444ae6482fa8f84572ddfa7bed" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44641 } } }
I20260620 19:19:56.876025 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6e59444ae6482fa8f84572ddfa7bed [sys.catalog]: This master's current role is: LEADER
I20260620 19:19:56.876127 15916 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:19:56.876797 15916 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:19:56.876883 15916 catalog_manager.cc:1259] Loaded cluster ID: a7f48a000cb74f1797bcea93a24a43ac
I20260620 19:19:56.876909 15916 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:19:56.877241 15916 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2855e1d9ad google::LogMessage::Fail() at ??:0
@ 0x7f2855e2088a google::LogMessage::SendToLog() at ??:0
@ 0x7f2855e1d4de google::LogMessage::Flush() at ??:0
@ 0x7f2855e1ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f285dd420f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f285dd4b8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2857ceb0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2857ce0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2858b446db start_thread at ??:0
@ 0x7f285491b61f clone at ??:0
W20260620 19:19:56.957726 15907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:56.957882 15907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:56.957902 15907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:56.958983 15907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:56.959023 15907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:56.959049 15907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:19:56.960356 15907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37879
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:56.960561 15907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:56.960774 15907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:56.965160 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:56.965173 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.965401 15907 server_base.cc:1034] running on GCE node
W20260620 19:19:56.965440 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:56.965643 15907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:56.965855 15907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:56.967015 15907 hybrid_clock.cc:648] HybridClock initialized: now 1781983196967003 us; error 26 us; skew 500 ppm
Jun 20 19:19:56 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:56.969308 15907 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:19:56.969710 15907 webserver.cc:460] Webserver started at http://127.1.1.193:37879/ using document root <none> and password file <none>
I20260620 19:19:56.969915 15907 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:56.969978 15907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:56.971118 15907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:19:56.971684 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:56.971817 15907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:19:56.971900 15907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "18c51fb6346ba7e32854f5d054e15969"
server_key_iv: "cf8fd89693ff7c4be81be89702283092"
server_key_version: "encryptionkey@0"
I20260620 19:19:56.972167 15907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:56.983726 15907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:56.985636 15907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:56.985777 15907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:56.985953 15907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:56.986306 15958 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:19:56.987922 15907 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:19:56.987978 15907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:19:56.988034 15907 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:19:56.988512 15907 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:19:56.988561 15907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:19:56.988672 15958 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap starting.
I20260620 19:19:56.995357 15907 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:42077
I20260620 19:19:56.995440 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:42077 every 8 connection(s)
I20260620 19:19:56.995692 15907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:19:56.998826 15958 log.cc:824] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:19:57.000566 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 15907
I20260620 19:19:57.000800 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:46617
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=39419
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:19:57.001012 15958 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:19:57.001297 15958 tablet_bootstrap.cc:492] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Bootstrap complete.
I20260620 19:19:57.002329 15958 ts_tablet_manager.cc:1397] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260620 19:19:57.003489 15958 raft_consensus.cc:348] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:57.003780 15958 raft_consensus.cc:729] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e659cdeb06a48bbb8e92ffff29bc9c9, State: Initialized, Role: FOLLOWER
I20260620 19:19:57.003893 15958 consensus_queue.cc:260] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:57.004117 15958 ts_tablet_manager.cc:1428] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260620 19:19:57.065527 16068 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:19:56.996337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:43911)
0620 19:19:56.996511 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:56.996538 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:56.996883 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:57.065357 (+ 68474us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
W20260620 19:19:57.065773 16066 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44641 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44641: Client connection negotiation failed: client connection to 127.1.1.254:44641: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:19:57.066078 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.065975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:57727)
0620 19:19:57.066029 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.066062 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:19:57.066355 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.066285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:45519)
0620 19:19:57.066324 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.066343 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:19:57.066463 16066 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:19:57.075770 16070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:57.075934 16070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:57.075976 16070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:57.076917 16070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:57.076972 16070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:57.077014 16070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:19:57.078269 16070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:46617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=39419
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:57.078498 16070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:57.078702 16070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:57.082855 16079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:57.082859 16077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:57.083000 16070 server_base.cc:1034] running on GCE node
W20260620 19:19:57.082876 16076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:57.083256 16070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:57.083458 16070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:57.084609 16070 hybrid_clock.cc:648] HybridClock initialized: now 1781983197084605 us; error 25 us; skew 500 ppm
Jun 20 19:19:57 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:57.086922 16070 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:19:57.087249 16070 webserver.cc:460] Webserver started at http://127.1.1.194:39419/ using document root <none> and password file <none>
I20260620 19:19:57.087425 16070 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:57.087479 16070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:57.088543 16070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:57.089030 16086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.089184 16070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.089269 16070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "40208add5a19dfbd282ba115152ced0c"
server_key_iv: "f31f6024776575ac2fd519a4b8fb1c5c"
server_key_version: "encryptionkey@0"
I20260620 19:19:57.089529 16070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:57.103870 16070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:57.105902 16070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:57.106029 16070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:57.106232 16070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:57.106483 16070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:57.106527 16070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.106559 16070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:57.106581 16070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.111330 16070 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:46617
I20260620 19:19:57.111395 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:46617 every 8 connection(s)
I20260620 19:19:57.111671 16070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:19:57.112558 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.112215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:42399)
0620 19:19:57.112343 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.112420 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
W20260620 19:19:57.112860 16200 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44641 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44641: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
I20260620 19:19:57.113122 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.113005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:57259)
0620 19:19:57.113050 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.113104 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:19:57.113360 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.113284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:40983)
0620 19:19:57.113332 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.113349 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:19:57.113448 16200 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:19:57.116354 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16070
I20260620 19:19:57.116568 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:45125
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:44641
--webserver_port=44505
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:41753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:19:57.186224 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:19:57.186386 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:19:57.186406 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:19:57.187461 16203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:19:57.187498 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:19:57.187522 16203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:19:57.188863 16203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:41753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:45125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=44505
--tserver_master_addrs=127.1.1.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:19:57.189064 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:19:57.189273 16203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:19:57.193465 16211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:57.193477 16208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:19:57.193514 16209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:19:57.193650 16203 server_base.cc:1034] running on GCE node
I20260620 19:19:57.193861 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:19:57.194048 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:19:57.195192 16203 hybrid_clock.cc:648] HybridClock initialized: now 1781983197195169 us; error 42 us; skew 500 ppm
Jun 20 19:19:57 dist-test-slave-mwkb krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:19:57.197335 16203 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:19:57.197690 16203 webserver.cc:460] Webserver started at http://127.1.1.195:44505/ using document root <none> and password file <none>
I20260620 19:19:57.197896 16203 fs_manager.cc:359] Metadata directory not provided
I20260620 19:19:57.197949 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:19:57.199149 16203 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:19:57.199700 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.199842 16203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:19:57.199923 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "00f842ed75294c849eb034964c91dd9b"
format_stamp: "Formatted at 2026-06-20 19:19:55 on dist-test-slave-mwkb"
server_key: "569eb6ee3ec0438f885f0ff349b020ff"
server_key_iv: "437359b99422bad73f50e016c64c2f98"
server_key_version: "encryptionkey@0"
I20260620 19:19:57.200222 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:19:57.211539 16203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:19:57.213500 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:19:57.213630 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:19:57.213851 16203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:19:57.214083 16203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:19:57.214130 16203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.214162 16203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:19:57.214205 16203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:19:57.219465 16203 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:45125
I20260620 19:19:57.219506 16331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:45125 every 8 connection(s)
I20260620 19:19:57.219777 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:19:57.220338 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16203
I20260620 19:19:57.220716 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.220341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:39591)
0620 19:19:57.220472 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.220560 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260620 19:19:57.221033 16332 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44641 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44641: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
I20260620 19:19:57.221324 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.221248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:46081)
0620 19:19:57.221286 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.221308 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:19:57.221535 16334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:57.221493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:49623)
0620 19:19:57.221510 (+ 17us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:57.221525 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10}
W20260620 19:19:57.221625 16332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:19:58.066931 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.066746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:36449)
0620 19:19:58.066849 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:58.066877 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260620 19:19:58.113900 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.113709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:55769)
0620 19:19:58.113826 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:58.113858 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260620 19:19:58.222054 16338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.221866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:41511)
0620 19:19:58.221982 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:58.222017 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":47,"threads_started":1}
I20260620 19:19:58.532677 16339 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:19:58.532788 16339 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:19:58.533170 16339 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
Jun 20 19:19:58 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.195@KRBTEST.COM
Jun 20 19:19:58 dist-test-slave-mwkb krb5kdc[14770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983196, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.194@KRBTEST.COM
I20260620 19:19:58.544070 16336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.533282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:45125 (local address 127.1.1.193:53709)
0620 19:19:58.533324 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:58.533349 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:58.533398 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:58.534225 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:58.534228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:58.534658 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:58.534833 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:58.534838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.535896 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.535897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:58.536647 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:58.536649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.537434 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.537435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:58.537544 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:58.538114 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:58.538121 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:58.539982 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:58.542295 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:58.542301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:58.542303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:58.542679 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:58.543046 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:58.543047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:58.543047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:58.543119 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:58.543537 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:58.543542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:58.543671 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:19:58.543788 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 19:19:58.543954 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:19:58.544225 16342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.533550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:56485 (local address 127.1.1.194:46617)
0620 19:19:58.533690 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:58.533696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:58.533718 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:19:58.534086 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:58.534087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:58.534117 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:58.534258 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:58.534985 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.536344 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.537000 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.537454 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.537541 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:58.540667 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:58.540670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:58.540670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:58.540678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:58.542296 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:58.542704 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:58.542707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:58.542708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:58.542813 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:58.543154 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:58.543155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:58.543156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:58.543648 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:58.543771 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:58.543939 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:19:58.544084 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":44,"threads_started":1}
I20260620 19:19:58.544736 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.533394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:53709 (local address 127.1.1.195:45125)
0620 19:19:58.533528 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 19:19:58.533539 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:19:58.533578 (+ 39us) server_negotiation.cc:407] Connection header received
0620 19:19:58.534022 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:19:58.534024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:19:58.534061 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:19:58.534159 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:19:58.534957 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.535806 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.536738 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.537346 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.537456 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:58.540079 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:19:58.540083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:19:58.540083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:19:58.540098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:19:58.542210 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:58.542769 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:58.542772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:58.542773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:58.542976 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:19:58.543209 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:19:58.543210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:19:58.543211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:19:58.543449 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:19:58.543544 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:19:58.544250 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 19:19:58.544610 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260620 19:19:58.545501 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:58.533331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:46617 (local address 127.1.1.193:56485)
0620 19:19:58.533492 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:58.533503 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:19:58.533557 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:19:58.534259 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:19:58.534260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:19:58.534686 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:19:58.534834 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:58.534841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.536430 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.536431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:58.536921 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:19:58.536922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:19:58.537613 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:19:58.537614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:19:58.537660 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:19:58.538311 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:19:58.538316 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:19:58.540565 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:19:58.542347 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:58.542348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:58.542349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:58.542584 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:58.542866 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:19:58.542866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:19:58.542867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:19:58.543062 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:19:58.543754 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:19:58.543755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:19:58.543792 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:19:58.543860 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:19:58.544029 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":99,"thread_start_us":62,"threads_started":1}
W20260620 19:19:58.545758 15954 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
W20260620 19:19:58.546131 15953 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:19:58.546188 15953 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters: 00f842ed75294c849eb034964c91dd9b, 0bcb98ca2a1348ccb5a4aaaf2c018936
I20260620 19:19:58.546316 16339 raft_consensus.cc:2726] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:19:59.067499 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:59.067308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:43373)
0620 19:19:59.067424 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:59.067459 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260620 19:19:59.114428 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:59.114255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:59765)
0620 19:19:59.114352 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:59.114399 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260620 19:19:59.222673 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:19:59.222417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:60155)
0620 19:19:59.222611 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:19:59.222641 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260620 19:20:00.067999 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:00.067818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:59489)
0620 19:20:00.067926 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:00.067957 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260620 19:20:00.114943 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:00.114766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:35815)
0620 19:20:00.114880 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:00.114913 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260620 19:20:00.223208 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:00.223038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:40277)
0620 19:20:00.223147 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:00.223177 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260620 19:20:00.678424 16349 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:00.678522 16349 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:20:00.678719 16349 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
W20260620 19:20:00.679071 15953 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
W20260620 19:20:00.679150 15954 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:20:00.679275 15954 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters: 00f842ed75294c849eb034964c91dd9b, 0bcb98ca2a1348ccb5a4aaaf2c018936
I20260620 19:20:00.679359 16349 raft_consensus.cc:2726] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:20:01.068638 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:01.068435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:48245)
0620 19:20:01.068568 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:01.068597 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260620 19:20:01.115491 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:01.115295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:47967)
0620 19:20:01.115428 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:01.115460 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260620 19:20:01.223737 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:01.223580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:58833)
0620 19:20:01.223676 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:01.223705 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":52,"threads_started":1}
I20260620 19:20:02.069399 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:02.069196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:52131)
0620 19:20:02.069335 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:02.069359 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 19:20:02.116056 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:02.115876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:59603)
0620 19:20:02.115997 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:02.116024 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 19:20:02.224272 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:02.224092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:38243)
0620 19:20:02.224209 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:02.224241 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":66,"threads_started":1}
I20260620 19:20:02.368337 16356 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:02.368408 16356 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:20:02.368546 16356 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
W20260620 19:20:02.368897 15954 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
W20260620 19:20:02.368988 15953 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:20:02.369020 15953 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters: 00f842ed75294c849eb034964c91dd9b, 0bcb98ca2a1348ccb5a4aaaf2c018936
I20260620 19:20:02.369118 16356 raft_consensus.cc:2726] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:20:03.069941 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:03.069741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:59621)
0620 19:20:03.069868 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:03.069899 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260620 19:20:03.116590 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:03.116438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:54967)
0620 19:20:03.116536 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:03.116563 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260620 19:20:03.224895 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:03.224671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:54993)
0620 19:20:03.224787 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:03.224849 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260620 19:20:04.070475 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:04.070293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:57853)
0620 19:20:04.070396 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:04.070426 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260620 19:20:04.117130 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:04.116931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:40733)
0620 19:20:04.117029 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:04.117059 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":45,"threads_started":1}
I20260620 19:20:04.225540 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:04.225343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:44139)
0620 19:20:04.225471 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:04.225503 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260620 19:20:05.071121 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:05.070923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:59349)
0620 19:20:05.071048 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:05.071079 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260620 19:20:05.117635 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:05.117478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:50831)
0620 19:20:05.117576 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:05.117604 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":52,"threads_started":1}
I20260620 19:20:05.226364 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:05.226157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:58015)
0620 19:20:05.226287 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:05.226321 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260620 19:20:06.071827 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:06.071632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:34787)
0620 19:20:06.071752 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:06.071783 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260620 19:20:06.118157 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:06.117991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:54335)
0620 19:20:06.118099 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:06.118129 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
I20260620 19:20:06.226956 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:06.226767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:36009)
0620 19:20:06.226884 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:06.226918 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 19:20:06.914229 16369 raft_consensus.cc:482] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:06.914330 16369 raft_consensus.cc:504] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f842ed75294c849eb034964c91dd9b" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 45125 } } peers { permanent_uuid: "0bcb98ca2a1348ccb5a4aaaf2c018936" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 46617 } } peers { permanent_uuid: "4e659cdeb06a48bbb8e92ffff29bc9c9" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 42077 } }
I20260620 19:20:06.914516 16369 leader_election.cc:288] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125), 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617)
W20260620 19:20:06.914841 15953 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0bcb98ca2a1348ccb5a4aaaf2c018936 (127.1.1.194:46617): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
W20260620 19:20:06.914913 15954 leader_election.cc:341] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 00f842ed75294c849eb034964c91dd9b (127.1.1.195:45125): Not found: Tablet not found: f6b750af5918483f9e84bb0ebe5d85ca
I20260620 19:20:06.914983 15954 leader_election.cc:302] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e659cdeb06a48bbb8e92ffff29bc9c9; no voters: 00f842ed75294c849eb034964c91dd9b, 0bcb98ca2a1348ccb5a4aaaf2c018936
I20260620 19:20:06.915048 16369 raft_consensus.cc:2726] T f6b750af5918483f9e84bb0ebe5d85ca P 4e659cdeb06a48bbb8e92ffff29bc9c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:20:07.072366 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:07.072174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:42583)
0620 19:20:07.072291 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:07.072324 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260620 19:20:07.118726 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:07.118555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:39187)
0620 19:20:07.118665 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:07.118695 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260620 19:20:07.227581 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:07.227388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:60043)
0620 19:20:07.227490 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:07.227526 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260620 19:20:08.072947 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:08.072766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:54083)
0620 19:20:08.072876 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:08.072906 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260620 19:20:08.119335 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:08.119125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:33863)
0620 19:20:08.119227 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:08.119259 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260620 19:20:08.228245 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:08.228061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:60481)
0620 19:20:08.228172 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:08.228205 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260620 19:20:09.073535 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:09.073351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:41609)
0620 19:20:09.073469 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:09.073496 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260620 19:20:09.120005 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:09.119743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:39667)
0620 19:20:09.119944 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:09.119975 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":59,"threads_started":1}
I20260620 19:20:09.229480 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:09.229017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:34331)
0620 19:20:09.229392 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:09.229421 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":335,"threads_started":1}
I20260620 19:20:10.074165 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:10.073976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:36577)
0620 19:20:10.074093 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:10.074123 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260620 19:20:10.120514 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:10.120352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:39843)
0620 19:20:10.120457 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:10.120485 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260620 19:20:10.230209 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:10.230044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:50843)
0620 19:20:10.230143 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:10.230173 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":54,"threads_started":1}
I20260620 19:20:11.074906 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:11.074673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:51743)
0620 19:20:11.074808 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:11.074847 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260620 19:20:11.121057 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:11.120877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:33451)
0620 19:20:11.120993 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:11.121024 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260620 19:20:11.230816 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:11.230648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:46461)
0620 19:20:11.230753 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:11.230782 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260620 19:20:12.075574 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:12.075387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.193:57907)
0620 19:20:12.075503 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:12.075534 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260620 19:20:12.121649 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:12.121498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.194:52003)
0620 19:20:12.121589 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:12.121619 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260620 19:20:12.226336 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 15907
I20260620 19:20:12.231004 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16070
I20260620 19:20:12.232285 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:12.231670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44641 (local address 127.1.1.195:59291)
0620 19:20:12.232205 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:12.232238 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":48,"threads_started":1}
I20260620 19:20:12.235062 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16203
2026-06-20T19:20:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17886 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:20:12 dist-test-slave-mwkb krb5kdc[16390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:20:12 dist-test-slave-mwkb krb5kdc[16390](info): set up 2 sockets
Jun 20 19:20:12 dist-test-slave-mwkb krb5kdc[16390](info): commencing operation
krb5kdc: starting...
W20260620 19:20:13.188843 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:20:13.289251 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:13.362681 16403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:13.362974 16403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:13.363015 16403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:13.364210 16403 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:20:13.364270 16403 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:20:13.364293 16403 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:20:13.364321 16403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:20:13.364344 16403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:13.364373 16403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:13.364406 16403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:20:13.364427 16403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:20:13.366010 16403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:13.366319 16403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:13.366551 16403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:20:13.371367 16403 server_base.cc:1034] running on GCE node
W20260620 19:20:13.371439 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.371488 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.371667 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:13.371901 16403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:13.372138 16403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:13.373271 16403 hybrid_clock.cc:648] HybridClock initialized: now 1781983213373255 us; error 30 us; skew 500 ppm
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:13.375597 16403 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:20:13.376010 16403 webserver.cc:460] Webserver started at http://127.1.1.254:45953/ using document root <none> and password file <none>
I20260620 19:20:13.376173 16403 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:13.376211 16403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:13.376291 16403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:20:13.377244 16403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "d785c3e619a04f1c810c3ecd29f53d69"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "31327c46deda41137911b74b135c9a99"
server_key_iv: "94f6c1e41c24ec13ac01487ed39c9fec"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.377532 16403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "d785c3e619a04f1c810c3ecd29f53d69"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "31327c46deda41137911b74b135c9a99"
server_key_iv: "94f6c1e41c24ec13ac01487ed39c9fec"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.378707 16403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:20:13.379312 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.379480 16403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:20:13.379560 16403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d785c3e619a04f1c810c3ecd29f53d69"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "31327c46deda41137911b74b135c9a99"
server_key_iv: "94f6c1e41c24ec13ac01487ed39c9fec"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.379612 16403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:13.407183 16403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:13.408942 16403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:13.409044 16403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:13.412863 16403 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44875
I20260620 19:20:13.412930 16470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44875 every 8 connection(s)
I20260620 19:20:13.413295 16403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:20:13.414112 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:13.416510 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap starting.
I20260620 19:20:13.417140 16471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:13.417433 16471 log.cc:824] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:13.418078 16471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: No bootstrap required, opened a new log
I20260620 19:20:13.419567 16471 raft_consensus.cc:348] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:13.419690 16471 raft_consensus.cc:374] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:13.419737 16471 raft_consensus.cc:729] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Initialized, Role: FOLLOWER
I20260620 19:20:13.419896 16471 consensus_queue.cc:260] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:13.419975 16471 raft_consensus.cc:388] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:20:13.420025 16471 raft_consensus.cc:482] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:20:13.420071 16471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:13.420562 16471 raft_consensus.cc:504] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:13.420645 16471 leader_election.cc:302] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d785c3e619a04f1c810c3ecd29f53d69; no voters:
I20260620 19:20:13.420827 16471 leader_election.cc:288] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:20:13.420904 16474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:20:13.421041 16474 raft_consensus.cc:686] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 LEADER]: Becoming Leader. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Running, Role: LEADER
I20260620 19:20:13.421061 16471 sys_catalog.cc:564] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:20:13.421325 16474 consensus_queue.cc:237] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:13.421913 16476 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d785c3e619a04f1c810c3ecd29f53d69. Latest consensus state: current_term: 1 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:13.422013 16476 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:13.422194 16483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:20:13.422317 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:13.422410 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:13.422681 16483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:20:13.423142 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16403
I20260620 19:20:13.423255 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:20:13.423449 1031 external_mini_cluster.cc:1428] Setting key 1b18566cf4f06b39533b9d613976b0b3
I20260620 19:20:13.424256 16483 catalog_manager.cc:1347] Generated new cluster ID: 2d41d9eef95147f3b218f0b8e325d9a5
I20260620 19:20:13.424305 16483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:13.434504 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.424079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.1.1.254:44875)
0620 19:20:13.424243 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.424252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.424280 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:20:13.424388 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.424394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.424448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.424584 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:13.425292 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.426726 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.427745 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.428362 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.428472 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.431108 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:13.431112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:13.431112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:13.431123 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:13.432954 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.433358 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.433361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.433361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.433457 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.433670 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.433670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.433671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.433940 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:13.434052 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.434175 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.434294 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260620 19:20:13.448546 16483 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:20:13.449013 16483 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:20:13.454043 16483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Generated new TSK 0
I20260620 19:20:13.454197 16483 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:20:13.480954 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:13.569267 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:13.569427 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:13.569451 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:13.570698 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:13.570744 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:13.570793 16497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:20:13.572055 16497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:13.572252 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:13.572444 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:13.576695 16503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.576706 16502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.576706 16505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:13.576992 16497 server_base.cc:1034] running on GCE node
I20260620 19:20:13.577201 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:13.577396 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:13.578534 16497 hybrid_clock.cc:648] HybridClock initialized: now 1781983213578523 us; error 25 us; skew 500 ppm
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:13.580631 16497 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:20:13.581012 16497 webserver.cc:460] Webserver started at http://127.1.1.193:37539/ using document root <none> and password file <none>
I20260620 19:20:13.581220 16497 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:13.581264 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:13.581385 16497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:20:13.582310 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "738c4c4efbc2490a9480395cc5ca784b"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "f443735b9eff5c16839166111fa63a68"
server_key_iv: "714cd79b5982b155e58f907a38ca379b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.582609 16497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "738c4c4efbc2490a9480395cc5ca784b"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "f443735b9eff5c16839166111fa63a68"
server_key_iv: "714cd79b5982b155e58f907a38ca379b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.583734 16497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:20:13.584523 16512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.584774 16497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:13.584846 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "738c4c4efbc2490a9480395cc5ca784b"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "f443735b9eff5c16839166111fa63a68"
server_key_iv: "714cd79b5982b155e58f907a38ca379b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.584892 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:13.605108 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:13.606629 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:13.606726 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:13.606870 16497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:13.607156 16497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:13.607204 16497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.607221 16497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:13.607261 16497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.611944 16497 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:39053
I20260620 19:20:13.612007 16625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:39053 every 8 connection(s)
I20260620 19:20:13.612253 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:20:13.615094 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16497
I20260620 19:20:13.615171 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:20:13.615355 1031 external_mini_cluster.cc:1428] Setting key de695971b4d5763ca9bb4c3b358c1042
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:13.620822 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.612857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:49409 (local address 127.1.1.254:44875)
0620 19:20:13.612899 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.612901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.613059 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:20:13.613488 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.613489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.613500 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.613547 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:13.614283 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.614731 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.615757 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.616028 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.616081 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.618404 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:13.618405 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:13.618406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:13.618415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:13.619594 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.619963 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.619964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.619965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.620017 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.620263 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.620264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.620264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.620367 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:13.620449 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.620672 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.620746 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:20:13.620965 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.612830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:49409)
0620 19:20:13.612948 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.612981 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.613388 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.613597 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.613600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.614054 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:13.614198 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.614202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.614815 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.614816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.615690 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.615692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.616092 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.616093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.616236 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.616661 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:13.616670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:13.618312 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:13.619681 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.619687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.619689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.619878 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.620092 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.620092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.620093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.620188 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.620443 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:13.620448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:13.620562 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.620652 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.620781 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":57,"threads_started":1}
I20260620 19:20:13.621397 16626 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:13.621479 16626 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:13.621636 16626 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:13.622776 16435 ts_manager.cc:194] Registered new tserver with Master: 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053)
I20260620 19:20:13.623597 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:49409
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:20:13.639325 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:13.709484 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:13.709637 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:13.709657 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:13.710587 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:13.710615 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:13.710637 16631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:20:13.711882 16631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:13.712039 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:13.712183 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:13.716511 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.716544 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.716544 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:13.716672 16631 server_base.cc:1034] running on GCE node
I20260620 19:20:13.716950 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:13.717244 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:13.718396 16631 hybrid_clock.cc:648] HybridClock initialized: now 1781983213718374 us; error 28 us; skew 500 ppm
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:13.720685 16631 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:20:13.721128 16631 webserver.cc:460] Webserver started at http://127.1.1.194:44135/ using document root <none> and password file <none>
I20260620 19:20:13.721378 16631 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:13.721428 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:13.721549 16631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:20:13.722438 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "af8862ccc3794783b68aedfea3485e71"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "589227c85ec57a1ff3e6e952672811e1"
server_key_iv: "2b24a9a3f16acc8510f54f6c62f8335b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.722752 16631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "af8862ccc3794783b68aedfea3485e71"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "589227c85ec57a1ff3e6e952672811e1"
server_key_iv: "2b24a9a3f16acc8510f54f6c62f8335b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.723840 16631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:20:13.724659 16646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.725000 16631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:13.725116 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "af8862ccc3794783b68aedfea3485e71"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "589227c85ec57a1ff3e6e952672811e1"
server_key_iv: "2b24a9a3f16acc8510f54f6c62f8335b"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.725191 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:13.738878 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:13.740453 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:13.740543 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:13.740676 16631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:13.740868 16631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:13.740903 16631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.740921 16631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:13.740932 16631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.746061 16631 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36807
I20260620 19:20:13.746116 16759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36807 every 8 connection(s)
I20260620 19:20:13.746390 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:13.753101 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16631
I20260620 19:20:13.753194 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:20:13.753384 1031 external_mini_cluster.cc:1428] Setting key 72b80de274ef5035d9ccc3784d023bcb
I20260620 19:20:13.753857 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.747037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:49599 (local address 127.1.1.254:44875)
0620 19:20:13.747077 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.747079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.747285 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:20:13.747654 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.747654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.747666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.747711 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:13.748414 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.748829 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.749655 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.749829 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.749877 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.751473 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:13.751474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:13.751474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:13.751480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:13.752452 (+ 972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.752750 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.752750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.752751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.752792 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.753024 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.753025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.753025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.753182 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:13.753253 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.753681 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.753766 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:20:13.753928 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.746995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:49599)
0620 19:20:13.747169 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.747203 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.747582 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.747757 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.747760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.748200 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:13.748344 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.748347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.748899 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.748900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.749591 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.749593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.749893 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.749893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.749970 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.750307 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:13.750313 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:13.751396 (+ 1083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:13.752512 (+ 1116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.752517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.752518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.752693 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.752847 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.752847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.752848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.752966 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.753273 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:13.753279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:13.753517 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.753637 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.753767 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260620 19:20:13.754360 16760 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:13.754474 16760 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:13.754647 16760 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:13.754973 16435 ts_manager.cc:194] Registered new tserver with Master: af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
I20260620 19:20:13.755393 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:49599
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:20:13.774051 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:13.844820 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:13.845010 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:13.845044 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:13.846061 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:13.846127 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:13.846171 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:20:13.847533 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:13.847762 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:13.847982 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:13.852290 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.852238 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:13.852238 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:13.852418 16765 server_base.cc:1034] running on GCE node
I20260620 19:20:13.852615 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:13.852834 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:13.853986 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781983213853961 us; error 56 us; skew 500 ppm
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:13.856292 16765 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:20:13.856725 16765 webserver.cc:460] Webserver started at http://127.1.1.195:41343/ using document root <none> and password file <none>
I20260620 19:20:13.856948 16765 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:13.857012 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:13.857191 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:20:13.858016 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "60b91ad058a44424bf30fdee66f38d75"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "d09398a655b0485497e6d680efaa1696"
server_key_iv: "f099a193efb3ba65a621e5abed900ab9"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.858314 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "60b91ad058a44424bf30fdee66f38d75"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "d09398a655b0485497e6d680efaa1696"
server_key_iv: "f099a193efb3ba65a621e5abed900ab9"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.859529 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:20:13.860167 16780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.860316 16765 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.860402 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "60b91ad058a44424bf30fdee66f38d75"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "d09398a655b0485497e6d680efaa1696"
server_key_iv: "f099a193efb3ba65a621e5abed900ab9"
server_key_version: "encryptionkey@0"
I20260620 19:20:13.860481 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:13.869161 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:13.871027 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:13.871160 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:13.871330 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:13.871567 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:13.871613 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.871646 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:13.871670 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:13.877094 16765 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34161
I20260620 19:20:13.877121 16893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34161 every 8 connection(s)
I20260620 19:20:13.877401 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:20:13.877561 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16765
I20260620 19:20:13.877655 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:20:13.877835 1031 external_mini_cluster.cc:1428] Setting key fab9b28c7f9a627ebdccfcaac5803cbc
Jun 20 19:20:13 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983213, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:13.884866 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.878009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:51561 (local address 127.1.1.254:44875)
0620 19:20:13.878045 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.878047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.878239 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:20:13.878813 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.878813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.878824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.878867 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:13.879729 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.880169 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.880979 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.881179 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.881223 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.882889 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:13.882890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:13.882890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:13.882896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:13.883925 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.884224 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.884225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.884225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.884267 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.884438 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.884439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.884439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.884519 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:13.884560 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.884761 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.884811 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:20:13.884980 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.877983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:51561)
0620 19:20:13.878139 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.878175 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.878721 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.878946 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.878951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.879496 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:13.879658 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.879662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.880238 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.880238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.880914 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.880915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.881236 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.881237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.881317 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.881645 (+ 328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:13.881651 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:13.882821 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:13.883986 (+ 1165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.883991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.883992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.884166 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.884316 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:13.884317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:13.884317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:13.884393 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:13.884564 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:13.884567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:13.884651 (+ 84us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.884735 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.884827 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260620 19:20:13.885460 16894 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:13.885524 16894 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:13.885625 16894 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:13.885905 16435 ts_manager.cc:194] Registered new tserver with Master: 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161)
I20260620 19:20:13.886284 16435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:51561
I20260620 19:20:13.889410 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:20:13.894981 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.890491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.1.1.254:44875)
0620 19:20:13.890547 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.890548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.890605 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:20:13.890647 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.890648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.890654 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.890691 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:13.891147 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.891530 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.892136 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.892289 (+ 153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.892328 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.893249 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:13.893250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:13.893251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:13.893256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:13.894218 (+ 962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.894489 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.894490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.894490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.894531 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:13.894655 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:13.894656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:13.894656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:13.894752 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:13.894807 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.894869 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.894915 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:20:13.896215 16435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:20:13.897763 16435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:20:13.903083 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34161 (local address 127.0.0.1:54680)
0620 19:20:13.899805 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.899835 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.899890 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.900262 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.900264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.900265 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:13.900319 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.900321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901159 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.901159 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.902131 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.902133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.902817 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.902886 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.902896 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.902982 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.903027 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260620 19:20:13.903088 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.1.1.195:34161)
0620 19:20:13.900144 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.900147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.900155 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:20:13.900183 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.900185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.900197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.900260 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:13.900366 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901031 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902218 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.902744 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902815 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.902860 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.902997 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.903025 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":50,"threads_started":1}
I20260620 19:20:13.903225 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.1.1.194:36807)
0620 19:20:13.900076 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.900080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.900184 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:20:13.900228 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.900229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.900243 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.900306 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:13.900481 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901394 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902201 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.902832 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902888 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.902931 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.903096 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.903133 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260620 19:20:13.903419 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36807 (local address 127.0.0.1:46062)
0620 19:20:13.900116 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.900123 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.900169 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.900375 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.900376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.900378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:13.900408 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.900409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901484 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.901485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.902120 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.902122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.902962 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.903016 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.903020 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.903302 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.903345 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":10,"thread_start_us":60,"threads_started":1}
I20260620 19:20:13.903748 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39053 (local address 127.0.0.1:47492)
0620 19:20:13.899992 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.900000 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.900043 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.900280 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.900281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.900283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:13.900326 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.900328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901474 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.901475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.902479 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.902481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.903604 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.903605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.903651 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.903653 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.903689 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.903710 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":25,"thread_start_us":53,"threads_started":1}
I20260620 19:20:13.903707 16828 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8699a6820843039417b0f8bd892118 (DEFAULT_TABLE table=test-table [id=397411dd67674c24b031db4489fedab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:13.903825 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.899838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.1.1.193:39053)
0620 19:20:13.899958 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.899962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.900038 (+ 76us) server_negotiation.cc:407] Connection header received
0620 19:20:13.900101 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.900102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.900115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.900205 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:13.900391 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.901367 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.902541 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.903467 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.903530 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.903578 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.903722 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.903762 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":74,"threads_started":1}
I20260620 19:20:13.903988 16828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8699a6820843039417b0f8bd892118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:13.904306 16560 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8699a6820843039417b0f8bd892118 (DEFAULT_TABLE table=test-table [id=397411dd67674c24b031db4489fedab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:13.904592 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8699a6820843039417b0f8bd892118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:13.906172 16908 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Bootstrap starting.
I20260620 19:20:13.906806 16908 tablet_bootstrap.cc:654] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:13.906849 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8699a6820843039417b0f8bd892118 (DEFAULT_TABLE table=test-table [id=397411dd67674c24b031db4489fedab3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:13.907064 16909 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap starting.
I20260620 19:20:13.907215 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8699a6820843039417b0f8bd892118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:13.907382 16908 log.cc:824] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:13.907675 16909 tablet_bootstrap.cc:654] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:13.908089 16909 log.cc:824] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:13.908756 16909 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: No bootstrap required, opened a new log
I20260620 19:20:13.908831 16909 ts_tablet_manager.cc:1397] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:20:13.909806 16912 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Bootstrap starting.
I20260620 19:20:13.910214 16908 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: No bootstrap required, opened a new log
I20260620 19:20:13.910290 16908 ts_tablet_manager.cc:1397] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:20:13.910421 16912 tablet_bootstrap.cc:654] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:13.910336 16909 raft_consensus.cc:348] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.910508 16909 raft_consensus.cc:374] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:13.910552 16909 raft_consensus.cc:729] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738c4c4efbc2490a9480395cc5ca784b, State: Initialized, Role: FOLLOWER
I20260620 19:20:13.910666 16909 consensus_queue.cc:260] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.910887 16912 log.cc:824] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:13.910918 16909 ts_tablet_manager.cc:1428] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:20:13.910930 16626 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:13.911494 16912 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: No bootstrap required, opened a new log
I20260620 19:20:13.911556 16912 ts_tablet_manager.cc:1397] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:20:13.912024 16908 raft_consensus.cc:348] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.912148 16908 raft_consensus.cc:374] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:13.912168 16908 raft_consensus.cc:729] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b91ad058a44424bf30fdee66f38d75, State: Initialized, Role: FOLLOWER
I20260620 19:20:13.912262 16908 consensus_queue.cc:260] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.912503 16908 ts_tablet_manager.cc:1428] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:20:13.912511 16894 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:13.913256 16912 raft_consensus.cc:348] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.913420 16912 raft_consensus.cc:374] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:13.913444 16912 raft_consensus.cc:729] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8862ccc3794783b68aedfea3485e71, State: Initialized, Role: FOLLOWER
I20260620 19:20:13.913542 16912 consensus_queue.cc:260] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.913769 16760 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:13.913771 16912 ts_tablet_manager.cc:1428] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:20:13.962080 16913 raft_consensus.cc:482] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:13.962169 16913 raft_consensus.cc:504] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.962520 16913 leader_election.cc:288] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
I20260620 19:20:13.965091 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.962744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:45549 (local address 127.1.1.194:36807)
0620 19:20:13.962786 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.962788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.962798 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:20:13.962822 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.962823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.962832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.962907 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:13.963062 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.963590 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964409 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.964780 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964825 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.964832 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.964981 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.965012 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:20:13.965111 16628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.962668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36807 (local address 127.1.1.193:45549)
0620 19:20:13.962717 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.962727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.962765 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.962920 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.962920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.962922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:13.962998 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.963000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.963657 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.963658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.964335 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.964336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.964845 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.964915 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.964919 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.964976 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.965013 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:20:13.965338 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.962771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:49825 (local address 127.1.1.195:34161)
0620 19:20:13.962815 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.962817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.962921 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:20:13.962964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.962966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.962979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.963032 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:13.963194 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.963676 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964554 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.964889 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964949 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.964958 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.965241 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.965288 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:20:13.965452 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.962668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34161 (local address 127.1.1.193:49825)
0620 19:20:13.962853 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.962862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:13.962914 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:13.963059 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:13.963059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:13.963061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:13.963119 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.963121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.963746 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.963746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.964469 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:13.964472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.964983 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.964984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:13.965045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.965049 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:20:13.965128 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:20:13.965376 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":10,"thread_start_us":87,"threads_started":1}
I20260620 19:20:13.965462 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8699a6820843039417b0f8bd892118" candidate_uuid: "738c4c4efbc2490a9480395cc5ca784b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8862ccc3794783b68aedfea3485e71" is_pre_election: true
I20260620 19:20:13.965572 16714 raft_consensus.cc:2454] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738c4c4efbc2490a9480395cc5ca784b in term 0.
I20260620 19:20:13.965773 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8699a6820843039417b0f8bd892118" candidate_uuid: "738c4c4efbc2490a9480395cc5ca784b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b91ad058a44424bf30fdee66f38d75" is_pre_election: true
I20260620 19:20:13.965885 16848 raft_consensus.cc:2454] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738c4c4efbc2490a9480395cc5ca784b in term 0.
I20260620 19:20:13.966084 16516 leader_election.cc:302] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b, af8862ccc3794783b68aedfea3485e71; no voters:
I20260620 19:20:13.966204 16913 raft_consensus.cc:2781] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:20:13.966274 16913 raft_consensus.cc:482] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:20:13.966305 16913 raft_consensus.cc:3037] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:13.966954 16913 raft_consensus.cc:504] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.967095 16913 leader_election.cc:288] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 1 election: Requested vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
I20260620 19:20:13.967283 16714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8699a6820843039417b0f8bd892118" candidate_uuid: "738c4c4efbc2490a9480395cc5ca784b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8862ccc3794783b68aedfea3485e71"
I20260620 19:20:13.967308 16848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8699a6820843039417b0f8bd892118" candidate_uuid: "738c4c4efbc2490a9480395cc5ca784b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b91ad058a44424bf30fdee66f38d75"
I20260620 19:20:13.967363 16848 raft_consensus.cc:3037] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:13.967360 16714 raft_consensus.cc:3037] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:13.968170 16848 raft_consensus.cc:2454] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738c4c4efbc2490a9480395cc5ca784b in term 1.
I20260620 19:20:13.968170 16714 raft_consensus.cc:2454] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738c4c4efbc2490a9480395cc5ca784b in term 1.
I20260620 19:20:13.968340 16514 leader_election.cc:302] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b91ad058a44424bf30fdee66f38d75, 738c4c4efbc2490a9480395cc5ca784b; no voters:
I20260620 19:20:13.968430 16913 raft_consensus.cc:2781] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:20:13.968593 16913 raft_consensus.cc:686] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 LEADER]: Becoming Leader. State: Replica: 738c4c4efbc2490a9480395cc5ca784b, State: Running, Role: LEADER
I20260620 19:20:13.968700 16913 consensus_queue.cc:237] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:13.969413 16435 catalog_manager.cc:5495] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b reported cstate change: term changed from 0 to 1, leader changed from <none> to 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193). New cstate: current_term: 1 leader_uuid: "738c4c4efbc2490a9480395cc5ca784b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } health_report { overall_health: UNKNOWN } } }
I20260620 19:20:13.976882 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.974728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.1.1.193:39053)
0620 19:20:13.974782 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:13.974784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:13.974792 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:20:13.974850 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:13.974851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:13.974866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:13.974932 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:20:13.975038 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.975428 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.976141 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:13.976339 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:13.976418 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:13.976583 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:20:13.976675 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:20:13.976779 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:13.976796 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 19:20:13.976812 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:20:13.978313 16714 raft_consensus.cc:1261] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Refusing update from remote peer 738c4c4efbc2490a9480395cc5ca784b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:20:13.978348 16848 raft_consensus.cc:1261] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Refusing update from remote peer 738c4c4efbc2490a9480395cc5ca784b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:20:13.978614 16913 consensus_queue.cc:1035] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:20:13.978756 16918 consensus_queue.cc:1035] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:20:13.981006 16921 mvcc.cc:205] Tried to move back new op lower bound from 7299003244452974592 to 7299003244416212992. Current Snapshot: MvccSnapshot[applied={T|T < 7299003244452974592}]
I20260620 19:20:13.981428 16923 mvcc.cc:205] Tried to move back new op lower bound from 7299003244452974592 to 7299003244416212992. Current Snapshot: MvccSnapshot[applied={T|T < 7299003244452974592}]
I20260620 19:20:13.983111 16434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:20:13.983210 16434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:20:13.983843 16434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Sending DeleteTablet for 3 replicas of tablet 5b8699a6820843039417b0f8bd892118
I20260620 19:20:13.984109 16828 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8699a6820843039417b0f8bd892118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:54680
I20260620 19:20:13.984123 16694 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8699a6820843039417b0f8bd892118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:46062
I20260620 19:20:13.984208 16560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8699a6820843039417b0f8bd892118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:47492
I20260620 19:20:13.984368 16932 tablet_replica.cc:330] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: stopping tablet replica
I20260620 19:20:13.984434 16931 tablet_replica.cc:330] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: stopping tablet replica
I20260620 19:20:13.984419 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16497
I20260620 19:20:13.984494 16932 raft_consensus.cc:2229] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:20:13.984503 16931 raft_consensus.cc:2229] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:20:13.984579 16931 raft_consensus.cc:2258] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:13.985605 16931 ts_tablet_manager.cc:1905] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:20:13.986815 16930 tablet_replica.cc:330] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: stopping tablet replica
I20260620 19:20:13.986897 16931 ts_tablet_manager.cc:1918] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:20:13.986923 16930 raft_consensus.cc:2229] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:20:13.986984 16931 log.cc:1196] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/5b8699a6820843039417b0f8bd892118
I20260620 19:20:13.987021 16930 raft_consensus.cc:2258] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:13.987267 16931 ts_tablet_manager.cc:1939] T 5b8699a6820843039417b0f8bd892118 P af8862ccc3794783b68aedfea3485e71: Deleting consensus metadata
I20260620 19:20:13.987442 16930 ts_tablet_manager.cc:1905] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:20:13.987586 16422 catalog_manager.cc:4841] TS af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): tablet 5b8699a6820843039417b0f8bd892118 (table test-table [id=397411dd67674c24b031db4489fedab3]) successfully deleted
I20260620 19:20:13.988581 16930 ts_tablet_manager.cc:1918] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:20:13.988655 16930 log.cc:1196] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/5b8699a6820843039417b0f8bd892118
I20260620 19:20:13.988867 16930 ts_tablet_manager.cc:1939] T 5b8699a6820843039417b0f8bd892118 P 60b91ad058a44424bf30fdee66f38d75: Deleting consensus metadata
I20260620 19:20:13.989248 16420 catalog_manager.cc:4841] TS 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): tablet 5b8699a6820843039417b0f8bd892118 (table test-table [id=397411dd67674c24b031db4489fedab3]) successfully deleted
W20260620 19:20:13.989516 16419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:39053 (error 108)
I20260620 19:20:13.989709 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16631
I20260620 19:20:13.989840 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:13.989750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39053 (local address 127.0.0.1:47506)
0620 19:20:13.989786 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:13.989811 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:20:13.990428 16419 catalog_manager.cc:4584] TS 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b8699a6820843039417b0f8bd892118: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39053: connect: Connection refused (error 111)
I20260620 19:20:13.994475 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16765
I20260620 19:20:13.998569 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16403
I20260620 19:20:34.003013 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--webserver_interface=127.1.1.254
--webserver_port=45953
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:34.078819 16934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:34.078994 16934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:34.079015 16934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:34.080123 16934 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:20:34.080160 16934 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:20:34.080173 16934 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:20:34.080184 16934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:20:34.080231 16934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:34.080250 16934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:34.080271 16934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:20:34.080293 16934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:20:34.081754 16934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=45953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:34.081986 16934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:34.082261 16934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:34.091101 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:34.091075 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.091084 16934 server_base.cc:1034] running on GCE node
W20260620 19:20:34.091145 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.091606 16934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:34.091877 16934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:34.093118 16934 hybrid_clock.cc:648] HybridClock initialized: now 1781983234093052 us; error 32 us; skew 500 ppm
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:34.095415 16934 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:20:34.095839 16934 webserver.cc:460] Webserver started at http://127.1.1.254:45953/ using document root <none> and password file <none>
I20260620 19:20:34.096043 16934 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:34.096114 16934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:34.097375 16934 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:20:34.098167 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.098351 16934 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:20:34.098457 16934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d785c3e619a04f1c810c3ecd29f53d69"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "31327c46deda41137911b74b135c9a99"
server_key_iv: "94f6c1e41c24ec13ac01487ed39c9fec"
server_key_version: "encryptionkey@0"
I20260620 19:20:34.098829 16934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:34.118495 16934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:34.120584 16934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:34.120723 16934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:34.124570 16934 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44875
I20260620 19:20:34.124604 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44875 every 8 connection(s)
I20260620 19:20:34.124964 16934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:20:34.126291 17003 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:20:34.126863 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap starting.
I20260620 19:20:34.127705 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 16934
I20260620 19:20:34.127967 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:39053
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=37539
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:20:34.135038 17003 log.cc:824] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:34.136760 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:20:34.137035 17003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap complete.
I20260620 19:20:34.139139 17003 raft_consensus.cc:348] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:34.139353 17003 raft_consensus.cc:729] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Initialized, Role: FOLLOWER
I20260620 19:20:34.139535 17003 consensus_queue.cc:260] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:34.139592 17003 raft_consensus.cc:388] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:20:34.139658 17003 raft_consensus.cc:482] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:20:34.139708 17003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:20:34.140623 17003 raft_consensus.cc:504] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:34.140720 17003 leader_election.cc:302] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d785c3e619a04f1c810c3ecd29f53d69; no voters:
I20260620 19:20:34.140862 17003 leader_election.cc:288] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:20:34.140892 17007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:20:34.140995 17007 raft_consensus.cc:686] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 LEADER]: Becoming Leader. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Running, Role: LEADER
I20260620 19:20:34.141112 17003 sys_catalog.cc:564] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:20:34.141177 17007 consensus_queue.cc:237] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:34.141407 17007 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d785c3e619a04f1c810c3ecd29f53d69. Latest consensus state: current_term: 2 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:34.141474 17007 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:34.141428 17008 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:34.141619 17008 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:34.141662 17013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:20:34.142411 17013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:20:34.142531 17013 catalog_manager.cc:1259] Loaded cluster ID: 2d41d9eef95147f3b218f0b8e325d9a5
I20260620 19:20:34.142745 17013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:20:34.143544 17013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:20:34.143721 17013 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Loaded TSK: 0
I20260620 19:20:34.153236 17013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Generated new TSK 1
I20260620 19:20:34.153323 17013 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:20:34.215947 17005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:34.216120 17005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:34.216140 17005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:34.217119 17005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:34.217152 17005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:34.217175 17005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:20:34.218501 17005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37539
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:34.218703 17005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:34.218948 17005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:34.224090 17031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:34.224116 17033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.224263 17005 server_base.cc:1034] running on GCE node
W20260620 19:20:34.224116 17030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.224556 17005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:34.224794 17005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:34.225955 17005 hybrid_clock.cc:648] HybridClock initialized: now 1781983234225923 us; error 51 us; skew 500 ppm
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:34.228392 17005 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:20:34.228852 17005 webserver.cc:460] Webserver started at http://127.1.1.193:37539/ using document root <none> and password file <none>
I20260620 19:20:34.229112 17005 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:34.229183 17005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:34.230480 17005 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:34.231194 17040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.231348 17005 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:34.231444 17005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "738c4c4efbc2490a9480395cc5ca784b"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "f443735b9eff5c16839166111fa63a68"
server_key_iv: "714cd79b5982b155e58f907a38ca379b"
server_key_version: "encryptionkey@0"
I20260620 19:20:34.231762 17005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:34.246690 17005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:34.248687 17005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:34.248823 17005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:34.249053 17005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:34.249517 17047 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:20:34.250864 17005 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:20:34.250916 17005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:20:34.250939 17005 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:20:34.251554 17005 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:20:34.251595 17005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:20:34.251634 17047 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap starting.
I20260620 19:20:34.258994 17005 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:39053
I20260620 19:20:34.259487 17005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:20:34.261322 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:39053 every 8 connection(s)
I20260620 19:20:34.262712 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17005
I20260620 19:20:34.263000 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:36807
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=44135
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:20:34.263082 17047 log.cc:824] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:34.265033 17047 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:20:34.265301 17047 tablet_bootstrap.cc:492] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap complete.
I20260620 19:20:34.266390 17047 ts_tablet_manager.cc:1397] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260620 19:20:34.267454 17047 raft_consensus.cc:348] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:34.267730 17047 raft_consensus.cc:729] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738c4c4efbc2490a9480395cc5ca784b, State: Initialized, Role: FOLLOWER
I20260620 19:20:34.267890 17047 consensus_queue.cc:260] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } }
I20260620 19:20:34.268147 17047 ts_tablet_manager.cc:1428] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:34.272825 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.260278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:39513 (local address 127.1.1.254:44875)
0620 19:20:34.261620 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.261629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.261661 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:20:34.261801 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.261808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.261866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.262009 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:34.262833 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.264286 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.265647 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.266270 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.266376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.268925 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:34.268928 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:34.268929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:34.268940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:34.270913 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.271437 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.271440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.271441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.271552 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.271838 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.271838 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.271839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.272087 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:34.272174 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.272521 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.272656 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":100,"threads_started":1}
I20260620 19:20:34.272825 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.260125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:39513)
0620 19:20:34.260331 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.260386 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.260942 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.262059 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.262063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.262378 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:34.262638 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.262644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.264417 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.264418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.265551 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.265554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.266362 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.266363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.266498 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.267023 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:34.267032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:34.268803 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:34.270998 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.271005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.271007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.271336 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.271636 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.271637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.271637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.271767 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.272168 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:34.272174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:34.272331 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.272443 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.272607 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
I20260620 19:20:34.273864 17155 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:34.273947 17155 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:34.274093 17155 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:34.274854 16967 ts_manager.cc:194] Registered new tserver with Master: 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053)
I20260620 19:20:34.276057 16967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:39513
I20260620 19:20:34.276959 17155 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:34.284696 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.275483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39053 (local address 127.0.0.1:39784)
0620 19:20:34.280810 (+ 5327us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.280846 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.280905 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.281197 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.281200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.281203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:34.281253 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.281254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.282455 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.282456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.283631 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.283634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.284509 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.284510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.284561 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.284569 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.284619 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.284649 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5299,"thread_start_us":71,"threads_started":1}
I20260620 19:20:34.284740 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.275724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.1.1.193:39053)
0620 19:20:34.281016 (+ 5292us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.281020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.281029 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:20:34.281056 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.281058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.281091 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.281180 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:34.281331 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.282361 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.283717 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.284433 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.284505 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.284552 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.284647 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.284680 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5265,"thread_start_us":63,"threads_started":1}
I20260620 19:20:34.285106 17089 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8699a6820843039417b0f8bd892118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:13 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39784
I20260620 19:20:34.285538 17164 tablet_replica.cc:330] stopping tablet replica
I20260620 19:20:34.285640 17164 raft_consensus.cc:2229] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:20:34.285699 17164 raft_consensus.cc:2258] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:34.285941 17164 ts_tablet_manager.cc:1905] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:20:34.287643 17164 ts_tablet_manager.cc:1918] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:20:34.287696 17164 log.cc:1196] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal/wals/5b8699a6820843039417b0f8bd892118
I20260620 19:20:34.287896 17164 ts_tablet_manager.cc:1939] T 5b8699a6820843039417b0f8bd892118 P 738c4c4efbc2490a9480395cc5ca784b: Deleting consensus metadata
I20260620 19:20:34.288336 16951 catalog_manager.cc:4841] TS 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053): tablet 5b8699a6820843039417b0f8bd892118 (table test-table [id=397411dd67674c24b031db4489fedab3]) successfully deleted
W20260620 19:20:34.346720 17160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:34.346918 17160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:34.346956 17160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:34.348023 17160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:34.348081 17160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:34.348124 17160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:20:34.349558 17160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:36807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=44135
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:34.349799 17160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:34.350011 17160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:34.354434 17172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:34.354491 17170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:34.354472 17169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.354593 17160 server_base.cc:1034] running on GCE node
I20260620 19:20:34.354779 17160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:34.355021 17160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:34.356173 17160 hybrid_clock.cc:648] HybridClock initialized: now 1781983234356159 us; error 26 us; skew 500 ppm
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:34.358502 17160 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:20:34.358925 17160 webserver.cc:460] Webserver started at http://127.1.1.194:44135/ using document root <none> and password file <none>
I20260620 19:20:34.359094 17160 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:34.359141 17160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:34.360314 17160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:34.360796 17179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.360975 17160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.361048 17160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "af8862ccc3794783b68aedfea3485e71"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "589227c85ec57a1ff3e6e952672811e1"
server_key_iv: "2b24a9a3f16acc8510f54f6c62f8335b"
server_key_version: "encryptionkey@0"
I20260620 19:20:34.361351 17160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:34.379432 17160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:34.380931 17160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:34.381026 17160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:34.381238 17160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:34.381461 17160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:34.381494 17160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.381512 17160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:34.381524 17160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.386618 17160 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36807
I20260620 19:20:34.386689 17292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36807 every 8 connection(s)
I20260620 19:20:34.386965 17160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:20:34.392174 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17160
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:34.392455 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:34161
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=41343
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:20:34.395612 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.387594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:39373 (local address 127.1.1.254:44875)
0620 19:20:34.387636 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.387638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.387782 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:20:34.388187 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.388188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.388199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.388245 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:34.389021 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.389584 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.390572 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.390868 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.390941 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.392873 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:34.392874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:34.392874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:34.392883 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:34.394303 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.394777 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.394777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.394778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.394824 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.395067 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.395067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.395067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.395154 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:34.395198 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.395490 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.395552 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:20:34.395872 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.387577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:39373)
0620 19:20:34.387698 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.387727 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.388092 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.388294 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.388296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.388782 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:34.388952 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.388955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.389691 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.389692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.390476 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.390478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.390948 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.390949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.391066 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.391430 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:34.391439 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:34.392782 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:34.394383 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.394389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.394391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.394700 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.394881 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.394881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.394882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.395007 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.395211 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:34.395216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:34.395370 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.395473 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.395628 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260620 19:20:34.396301 17293 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:34.396396 17293 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:34.396557 17293 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:34.396924 16967 ts_manager.cc:194] Registered new tserver with Master: af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
I20260620 19:20:34.397475 16967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:39373
W20260620 19:20:34.468828 17296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:34.469002 17296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:34.469023 17296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:34.470088 17296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:34.470124 17296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:34.470148 17296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:20:34.471393 17296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:34161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=41343
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:34.471537 17296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:34.471673 17296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:34.475992 17302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:34.475996 17304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.476114 17296 server_base.cc:1034] running on GCE node
W20260620 19:20:34.476244 17301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:34.476460 17296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:34.476708 17296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:34.477861 17296 hybrid_clock.cc:648] HybridClock initialized: now 1781983234477844 us; error 28 us; skew 500 ppm
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:34.480131 17296 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:20:34.480540 17296 webserver.cc:460] Webserver started at http://127.1.1.195:41343/ using document root <none> and password file <none>
I20260620 19:20:34.480734 17296 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:34.480783 17296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:34.482030 17296 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:34.482537 17311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.482707 17296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:20:34.482788 17296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "60b91ad058a44424bf30fdee66f38d75"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "d09398a655b0485497e6d680efaa1696"
server_key_iv: "f099a193efb3ba65a621e5abed900ab9"
server_key_version: "encryptionkey@0"
I20260620 19:20:34.483067 17296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:34.499079 17296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:34.501256 17296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:34.501405 17296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:34.501616 17296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:34.501916 17296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:34.501950 17296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.501969 17296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:34.501982 17296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.507591 17296 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34161
I20260620 19:20:34.507634 17424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34161 every 8 connection(s)
I20260620 19:20:34.507916 17296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:20:34 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983234, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:20:34.516471 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.508525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:41397 (local address 127.1.1.254:44875)
0620 19:20:34.508585 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.508587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.508810 (+ 223us) server_negotiation.cc:407] Connection header received
0620 19:20:34.509298 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.509299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.509310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.509362 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:34.510223 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.510690 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.511616 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.511823 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.511899 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.513894 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:34.513896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:34.513896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:34.513903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:34.515088 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.515521 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.515522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.515522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.515581 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.515864 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.515865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.515866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.515976 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:34.516050 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.516309 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.516384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:20:34.516629 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.508501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:41397)
0620 19:20:34.508681 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.508727 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.509189 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.509432 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.509435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.509949 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:34.510115 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.510118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.510780 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.510782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.511525 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.511527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.511904 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.511906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.512027 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.512427 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:34.512434 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:34.513755 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:34.515196 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.515202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.515204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.515431 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.515646 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:34.515647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:34.515647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:34.515761 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:34.516043 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:34.516048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:34.516178 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.516281 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.516423 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260620 19:20:34.517141 17425 heartbeater.cc:347] Connected to a master server at 127.1.1.254:44875
I20260620 19:20:34.517223 17425 heartbeater.cc:464] Registering TS with master...
I20260620 19:20:34.517407 17425 heartbeater.cc:511] Master 127.1.1.254:44875 requested a full tablet report, sending...
I20260620 19:20:34.517490 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17296
I20260620 19:20:34.517804 16967 ts_manager.cc:194] Registered new tserver with Master: 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161)
I20260620 19:20:34.518170 16967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:41397
I20260620 19:20:34.524041 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.517832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.1.1.254:44875)
0620 19:20:34.517865 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.517866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.518060 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:20:34.518132 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.518133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.518143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.518197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:34.518856 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.519307 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.520057 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.520259 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.520342 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.521547 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:34.521548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:34.521549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:34.521555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:34.522922 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.523287 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.523288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.523289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.523339 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.523558 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.523559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.523559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.523670 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:34.523841 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.523878 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.523946 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:20:34.524750 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:20:34.527671 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.525200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.1.1.254:44875)
0620 19:20:34.525254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.525257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.525362 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:20:34.525442 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.525443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.525482 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.525556 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:20:34.525655 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.526113 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.526848 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.527026 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.527107 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.527286 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:20:34.527328 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 19:20:34.527599 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37236: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 19:20:34.527720 17158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37236: token expired
W20260620 19:20:34.528389 16897 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.1.254:44875)
I20260620 19:20:34.535687 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.528649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.1.1.254:44875)
0620 19:20:34.528703 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.528704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.528763 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:20:34.528847 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.528848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.528857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.528919 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:34.529475 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.530109 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.531345 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.531607 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.531676 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.533183 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:34.533184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:34.533185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:34.533191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:34.534743 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.535090 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.535091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.535091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.535135 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:34.535259 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:34.535259 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:34.535259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:34.535362 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:34.535425 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.535510 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.535580 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:20:34.536640 16967 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:20:34.538183 16967 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:20:34.540313 17089 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 (DEFAULT_TABLE table=test-table [id=21c6c69a6e67416ea70fead874ca406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:34.540500 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae5e8246164d04a0fd972d8ed31cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:34.542196 17047 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap starting.
I20260620 19:20:34.542548 17047 tablet_bootstrap.cc:654] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:34.543191 17047 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: No bootstrap required, opened a new log
I20260620 19:20:34.543243 17047 ts_tablet_manager.cc:1397] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:34.543349 17047 raft_consensus.cc:348] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.543430 17047 raft_consensus.cc:374] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:34.543509 17047 raft_consensus.cc:729] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738c4c4efbc2490a9480395cc5ca784b, State: Initialized, Role: FOLLOWER
I20260620 19:20:34.543591 17047 consensus_queue.cc:260] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.543684 17047 ts_tablet_manager.cc:1428] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:34.543751 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.539873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36807 (local address 127.0.0.1:54368)
0620 19:20:34.540006 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.540017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.540067 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.540397 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.540398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.540399 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:34.540443 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.540444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.541547 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.541548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.542414 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.542417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.543504 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.543505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.543585 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.543589 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.543643 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.543677 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260620 19:20:34.543980 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.540083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.1.1.194:36807)
0620 19:20:34.540248 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.540252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.540260 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:20:34.540291 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.540293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.540304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.540399 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:34.540498 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.541470 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.542505 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.543349 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.543420 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.543483 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.543853 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.543897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260620 19:20:34.544623 17227 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 (DEFAULT_TABLE table=test-table [id=21c6c69a6e67416ea70fead874ca406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:34.544746 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.539842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34161 (local address 127.0.0.1:37920)
0620 19:20:34.539887 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.539897 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.539932 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.541399 (+ 1467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.541400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.541402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:34.541439 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.541440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.542698 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.542699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.543708 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.543710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.544523 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.544524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.544585 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.544588 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.544644 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.544683 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:20:34.544952 17227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae5e8246164d04a0fd972d8ed31cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:34.544965 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.539914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37920 (local address 127.1.1.195:34161)
0620 19:20:34.541224 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.541235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.541246 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:20:34.541281 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.541283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.541297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.541510 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:34.541590 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.542585 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.543778 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.544443 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.544505 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.544537 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.544844 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.544885 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1251,"thread_start_us":118,"threads_started":1}
I20260620 19:20:34.545557 17359 tablet_service.cc:1467] Processing CreateTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 (DEFAULT_TABLE table=test-table [id=21c6c69a6e67416ea70fead874ca406f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:20:34.545868 17359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ae5e8246164d04a0fd972d8ed31cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:20:34.547817 17434 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Bootstrap starting.
I20260620 19:20:34.548537 17434 tablet_bootstrap.cc:654] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:34.548524 17435 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Bootstrap starting.
I20260620 19:20:34.549167 17435 tablet_bootstrap.cc:654] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Neither blocks nor log segments found. Creating new log.
I20260620 19:20:34.549382 17434 log.cc:824] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:34.549679 17435 log.cc:824] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:34.550094 17434 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: No bootstrap required, opened a new log
I20260620 19:20:34.550161 17434 ts_tablet_manager.cc:1397] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:20:34.550397 17435 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: No bootstrap required, opened a new log
I20260620 19:20:34.550447 17435 ts_tablet_manager.cc:1397] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:20:34.551548 17434 raft_consensus.cc:348] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.551682 17434 raft_consensus.cc:374] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:34.551723 17434 raft_consensus.cc:729] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8862ccc3794783b68aedfea3485e71, State: Initialized, Role: FOLLOWER
I20260620 19:20:34.551635 17435 raft_consensus.cc:348] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.551797 17435 raft_consensus.cc:374] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:20:34.551841 17435 raft_consensus.cc:729] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b91ad058a44424bf30fdee66f38d75, State: Initialized, Role: FOLLOWER
I20260620 19:20:34.551844 17434 consensus_queue.cc:260] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.551982 17435 consensus_queue.cc:260] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.552074 17434 ts_tablet_manager.cc:1428] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:20:34.552105 17293 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:34.552238 17435 ts_tablet_manager.cc:1428] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:20:34.552258 17425 heartbeater.cc:503] Master 127.1.1.254:44875 was elected leader, sending a full tablet report...
I20260620 19:20:34.557190 17439 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:34.557257 17439 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.557467 17439 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807), 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053)
I20260620 19:20:34.560137 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.557695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:57823 (local address 127.1.1.194:36807)
0620 19:20:34.557734 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.557736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.557744 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:20:34.557765 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.557766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.557776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.557853 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:34.557991 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.558472 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559436 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.559805 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559867 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.559898 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.560040 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.560078 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:20:34.560143 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.557597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36807 (local address 127.1.1.195:57823)
0620 19:20:34.557650 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.557660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.557694 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.557859 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.557860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.557862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:34.557925 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.557926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.558544 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.558545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.559351 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.559353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.559890 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.559972 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.559975 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.560017 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.560071 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:20:34.560179 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.557760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40485 (local address 127.1.1.193:39053)
0620 19:20:34.557797 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.557799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.557806 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:20:34.557878 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.557879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.557888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.557940 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:20:34.558088 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.558557 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559463 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.559819 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559902 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.559911 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.560076 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.560116 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:20:34.560495 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae5e8246164d04a0fd972d8ed31cd6" candidate_uuid: "60b91ad058a44424bf30fdee66f38d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8862ccc3794783b68aedfea3485e71" is_pre_election: true
I20260620 19:20:34.560537 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.557597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39053 (local address 127.1.1.195:40485)
0620 19:20:34.557764 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.557774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:34.557818 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:34.557941 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:34.557942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:34.557943 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:20:34.558023 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.558024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.558632 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.558633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.559358 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:34.559360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.559917 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.559918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:34.559984 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.559987 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:20:34.560060 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:20:34.560093 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260620 19:20:34.560621 17247 raft_consensus.cc:2454] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b91ad058a44424bf30fdee66f38d75 in term 0.
I20260620 19:20:34.560824 17315 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71; no voters:
I20260620 19:20:34.560814 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae5e8246164d04a0fd972d8ed31cd6" candidate_uuid: "60b91ad058a44424bf30fdee66f38d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738c4c4efbc2490a9480395cc5ca784b" is_pre_election: true
I20260620 19:20:34.560947 17109 raft_consensus.cc:2454] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b91ad058a44424bf30fdee66f38d75 in term 0.
I20260620 19:20:34.560968 17439 raft_consensus.cc:2781] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:20:34.561024 17439 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:20:34.561059 17439 raft_consensus.cc:3037] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:34.561635 17439 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.561753 17439 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [CANDIDATE]: Term 1 election: Requested vote from peers af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807), 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053)
I20260620 19:20:34.561874 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae5e8246164d04a0fd972d8ed31cd6" candidate_uuid: "60b91ad058a44424bf30fdee66f38d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8862ccc3794783b68aedfea3485e71"
I20260620 19:20:34.561942 17247 raft_consensus.cc:3037] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:34.561962 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4ae5e8246164d04a0fd972d8ed31cd6" candidate_uuid: "60b91ad058a44424bf30fdee66f38d75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738c4c4efbc2490a9480395cc5ca784b"
I20260620 19:20:34.562032 17109 raft_consensus.cc:3037] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:20:34.562531 17247 raft_consensus.cc:2454] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b91ad058a44424bf30fdee66f38d75 in term 1.
I20260620 19:20:34.562695 17315 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71; no voters:
I20260620 19:20:34.562791 17439 raft_consensus.cc:2781] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:20:34.562906 17439 raft_consensus.cc:686] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 1 LEADER]: Becoming Leader. State: Replica: 60b91ad058a44424bf30fdee66f38d75, State: Running, Role: LEADER
I20260620 19:20:34.562978 17439 consensus_queue.cc:237] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:34.563068 17109 raft_consensus.cc:2454] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b91ad058a44424bf30fdee66f38d75 in term 1.
I20260620 19:20:34.563575 16965 catalog_manager.cc:5495] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "60b91ad058a44424bf30fdee66f38d75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } health_report { overall_health: UNKNOWN } } }
I20260620 19:20:34.572039 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.569851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.1.1.195:34161)
0620 19:20:34.569884 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:34.569886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:34.569902 (+ 16us) server_negotiation.cc:407] Connection header received
0620 19:20:34.569926 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:34.569927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:34.569948 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:34.570008 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:20:34.570103 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.570494 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.571311 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:34.571500 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:34.571575 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:34.571732 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:20:34.571817 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:20:34.571904 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:34.571927 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:20:34.571951 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:20:34.573395 17109 raft_consensus.cc:1261] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Refusing update from remote peer 60b91ad058a44424bf30fdee66f38d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:20:34.573395 17247 raft_consensus.cc:1261] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Refusing update from remote peer 60b91ad058a44424bf30fdee66f38d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:20:34.573755 17441 consensus_queue.cc:1035] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:20:34.573853 17439 consensus_queue.cc:1035] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:20:34.575722 17445 mvcc.cc:205] Tried to move back new op lower bound from 7299003328810598400 to 7299003328770543616. Current Snapshot: MvccSnapshot[applied={T|T < 7299003328810598400}]
I20260620 19:20:34.575857 17444 mvcc.cc:205] Tried to move back new op lower bound from 7299003328810598400 to 7299003328770543616. Current Snapshot: MvccSnapshot[applied={T|T < 7299003328810598400}]
I20260620 19:20:34.578148 16966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:20:34.578263 16966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:20:34.578902 16966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Sending DeleteTablet for 3 replicas of tablet c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:20:34.579200 17227 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:34 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:54368
I20260620 19:20:34.579259 17089 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:34 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:39784
I20260620 19:20:34.579380 17164 tablet_replica.cc:330] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: stopping tablet replica
I20260620 19:20:34.579419 17454 tablet_replica.cc:330] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: stopping tablet replica
I20260620 19:20:34.579449 17164 raft_consensus.cc:2229] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:20:34.579509 17164 raft_consensus.cc:2258] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:34.579505 17454 raft_consensus.cc:2229] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:20:34.579592 17454 raft_consensus.cc:2258] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:34.579700 17359 tablet_service.cc:1514] Processing DeleteTablet for tablet c4ae5e8246164d04a0fd972d8ed31cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:20:34 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:37920
I20260620 19:20:34.579944 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17005
I20260620 19:20:34.580058 17454 ts_tablet_manager.cc:1905] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:20:34.581676 17454 ts_tablet_manager.cc:1918] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:20:34.581741 17454 log.cc:1196] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:20:34.581979 17454 ts_tablet_manager.cc:1939] T c4ae5e8246164d04a0fd972d8ed31cd6 P af8862ccc3794783b68aedfea3485e71: Deleting consensus metadata
I20260620 19:20:34.582316 17455 tablet_replica.cc:330] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: stopping tablet replica
I20260620 19:20:34.582355 16954 catalog_manager.cc:4841] TS af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): tablet c4ae5e8246164d04a0fd972d8ed31cd6 (table test-table [id=21c6c69a6e67416ea70fead874ca406f]) successfully deleted
I20260620 19:20:34.582401 17455 raft_consensus.cc:2229] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:20:34.582494 17455 raft_consensus.cc:2258] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:20:34.582837 17455 ts_tablet_manager.cc:1905] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:20:34.584530 17455 ts_tablet_manager.cc:1918] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:20:34.584592 17455 log.cc:1196] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:20:34.584820 17455 ts_tablet_manager.cc:1939] T c4ae5e8246164d04a0fd972d8ed31cd6 P 60b91ad058a44424bf30fdee66f38d75: Deleting consensus metadata
I20260620 19:20:34.585198 16952 catalog_manager.cc:4841] TS 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): tablet c4ae5e8246164d04a0fd972d8ed31cd6 (table test-table [id=21c6c69a6e67416ea70fead874ca406f]) successfully deleted
W20260620 19:20:34.586640 16951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:39053 (error 108)
I20260620 19:20:34.586925 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17160
I20260620 19:20:34.587477 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:34.587025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:39053 (local address 127.0.0.1:39790)
0620 19:20:34.587416 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:34.587446 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
W20260620 19:20:34.587612 16951 catalog_manager.cc:4584] TS 738c4c4efbc2490a9480395cc5ca784b (127.1.1.193:39053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4ae5e8246164d04a0fd972d8ed31cd6: Network error: Client connection negotiation failed: client connection to 127.1.1.193:39053: connect: Connection refused (error 111)
I20260620 19:20:34.593461 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17296
I20260620 19:20:34.597162 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 16934
I20260620 19:20:54.603209 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--webserver_interface=127.1.1.254
--webserver_port=45953
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:54.678987 17457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:54.679195 17457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:54.679215 17457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:54.680315 17457 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:20:54.680353 17457 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:20:54.680366 17457 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:20:54.680377 17457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:20:54.680431 17457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:54.680449 17457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:54.680472 17457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:20:54.680503 17457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:20:54.681893 17457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=45953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:54.682081 17457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:54.682294 17457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:54.686887 17465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:54.686950 17457 server_base.cc:1034] running on GCE node
W20260620 19:20:54.686965 17462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:54.687093 17463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:54.687299 17457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:54.687587 17457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:54.688767 17457 hybrid_clock.cc:648] HybridClock initialized: now 1781983254688740 us; error 43 us; skew 500 ppm
Jun 20 19:20:54 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:54.691267 17457 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:20:54.691606 17457 webserver.cc:460] Webserver started at http://127.1.1.254:45953/ using document root <none> and password file <none>
I20260620 19:20:54.691758 17457 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:54.691792 17457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:54.692830 17457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:20:54.693439 17472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:54.693594 17457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:54.693663 17457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d785c3e619a04f1c810c3ecd29f53d69"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "31327c46deda41137911b74b135c9a99"
server_key_iv: "94f6c1e41c24ec13ac01487ed39c9fec"
server_key_version: "encryptionkey@0"
I20260620 19:20:54.693939 17457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:54.706547 17457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:54.708062 17457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:54.708189 17457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:54.711916 17457 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44875
I20260620 19:20:54.711983 17524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44875 every 8 connection(s)
I20260620 19:20:54.712293 17457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:20:54.713492 17525 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:20:54.714012 17525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap starting.
I20260620 19:20:54.717813 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17457
I20260620 19:20:54.718050 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:39053
--local_ip_for_outbound_sockets=127.1.1.193
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=37539
--webserver_interface=127.1.1.193
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:20:54.722227 17525 log.cc:824] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:54.724014 17525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:20:54.724274 17525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: Bootstrap complete.
I20260620 19:20:54.726186 17525 raft_consensus.cc:348] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:54.726409 17525 raft_consensus.cc:729] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Initialized, Role: FOLLOWER
I20260620 19:20:54.726580 17525 consensus_queue.cc:260] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:54.726660 17525 raft_consensus.cc:388] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:20:54.726706 17525 raft_consensus.cc:482] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:20:54.726745 17525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:20:54.727595 17525 raft_consensus.cc:504] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:54.727701 17525 leader_election.cc:302] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d785c3e619a04f1c810c3ecd29f53d69; no voters:
I20260620 19:20:54.727876 17525 leader_election.cc:288] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:20:54.727905 17529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:20:54.728009 17529 raft_consensus.cc:686] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [term 3 LEADER]: Becoming Leader. State: Replica: d785c3e619a04f1c810c3ecd29f53d69, State: Running, Role: LEADER
I20260620 19:20:54.728102 17525 sys_catalog.cc:564] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:20:54.728083 17529 consensus_queue.cc:237] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } }
I20260620 19:20:54.728324 17530 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:54.728330 17531 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d785c3e619a04f1c810c3ecd29f53d69. Latest consensus state: current_term: 3 leader_uuid: "d785c3e619a04f1c810c3ecd29f53d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785c3e619a04f1c810c3ecd29f53d69" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44875 } } }
I20260620 19:20:54.728466 17530 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:54.728482 17531 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69 [sys.catalog]: This master's current role is: LEADER
I20260620 19:20:54.729794 17545 catalog_manager.cc:1259] Loaded cluster ID: 2d41d9eef95147f3b218f0b8e325d9a5
I20260620 19:20:54.729839 17545 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: loading cluster ID for follower catalog manager: success
I20260620 19:20:54.730916 17545 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d785c3e619a04f1c810c3ecd29f53d69: acquiring CA information for follower catalog manager: success
F20260620 19:20:54.731262 17545 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9845e9f9ad google::LogMessage::Fail() at ??:0
@ 0x7f9845ea288a google::LogMessage::SendToLog() at ??:0
@ 0x7f9845e9f4de google::LogMessage::Flush() at ??:0
@ 0x7f9845ea0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f984860c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f984ddfc6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f984ddfc989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f984ddfce4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f984ddff8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f9847d62954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9848bc66db start_thread at ??:0
@ 0x7f984499d61f clone at ??:0
W20260620 19:20:54.819968 17527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:54.820205 17527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:54.820246 17527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:54.821887 17527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:54.821941 17527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:54.821975 17527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:20:54.823839 17527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:39053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=37539
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:54.824052 17527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:54.824262 17527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:20:54.829003 17527 server_base.cc:1034] running on GCE node
W20260620 19:20:54.829183 17564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:54.829237 17563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:54.829344 17566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:54.829506 17527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:54.829731 17527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:54.830924 17527 hybrid_clock.cc:648] HybridClock initialized: now 1781983254830908 us; error 29 us; skew 500 ppm
Jun 20 19:20:54 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:54.838424 17527 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:20:54.838881 17527 webserver.cc:460] Webserver started at http://127.1.1.193:37539/ using document root <none> and password file <none>
I20260620 19:20:54.839183 17527 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:54.839264 17527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:54.840989 17527 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:20:54.841709 17573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:54.841887 17527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:20:54.841974 17527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "738c4c4efbc2490a9480395cc5ca784b"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "f443735b9eff5c16839166111fa63a68"
server_key_iv: "714cd79b5982b155e58f907a38ca379b"
server_key_version: "encryptionkey@0"
I20260620 19:20:54.842336 17527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:54.869545 17527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:54.871759 17527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:54.871893 17527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:54.872117 17527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:54.872581 17580 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:20:54.873987 17527 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:20:54.874048 17527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:20:54.874081 17527 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:20:54.874888 17527 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:20:54.874930 17527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:20:54.874989 17580 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap starting.
I20260620 19:20:54.881788 17527 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:39053
I20260620 19:20:54.882016 17687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:39053 every 8 connection(s)
I20260620 19:20:54.882195 17527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:20:54.883522 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17527
I20260620 19:20:54.883818 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:36807
--local_ip_for_outbound_sockets=127.1.1.194
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=44135
--webserver_interface=127.1.1.194
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:20:54.885705 17580 log.cc:824] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:20:54.887512 17580 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:20:54.887805 17580 tablet_bootstrap.cc:492] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Bootstrap complete.
I20260620 19:20:54.888931 17580 ts_tablet_manager.cc:1397] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260620 19:20:54.890758 17580 raft_consensus.cc:348] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:54.891134 17580 raft_consensus.cc:729] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738c4c4efbc2490a9480395cc5ca784b, State: Initialized, Role: FOLLOWER
I20260620 19:20:54.891315 17580 consensus_queue.cc:260] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:54.891634 17580 ts_tablet_manager.cc:1428] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
W20260620 19:20:54.936654 17690 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:20:54.882848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:39275)
0620 19:20:54.883066 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:54.883118 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:54.883715 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:54.936459 (+ 52744us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
W20260620 19:20:54.936918 17688 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44875 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44875: Client connection negotiation failed: client connection to 127.1.1.254:44875: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:20:54.937317 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:54.937185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:52371)
0620 19:20:54.937246 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:54.937298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:20:54.937641 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:54.937560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:40563)
0620 19:20:54.937604 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:54.937627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:20:54.937772 17688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:20:54.965579 17691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:54.965757 17691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:54.965792 17691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:54.966851 17691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:54.966923 17691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:54.966979 17691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:20:54.968250 17691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:36807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=44135
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:54.968472 17691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:54.968658 17691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:54.973278 17701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:54.973280 17699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:54.973366 17691 server_base.cc:1034] running on GCE node
W20260620 19:20:54.973296 17698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:54.973660 17691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:54.973914 17691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:54.975051 17691 hybrid_clock.cc:648] HybridClock initialized: now 1781983254975038 us; error 23 us; skew 500 ppm
Jun 20 19:20:54 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:54.977120 17691 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:20:54.977516 17691 webserver.cc:460] Webserver started at http://127.1.1.194:44135/ using document root <none> and password file <none>
I20260620 19:20:54.977705 17691 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:54.977746 17691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:54.978782 17691 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:54.979293 17708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:54.979465 17691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:20:54.979537 17691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "af8862ccc3794783b68aedfea3485e71"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "589227c85ec57a1ff3e6e952672811e1"
server_key_iv: "2b24a9a3f16acc8510f54f6c62f8335b"
server_key_version: "encryptionkey@0"
I20260620 19:20:54.979800 17691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:55.001302 17691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:55.002799 17691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:55.002892 17691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:55.003038 17691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:55.003232 17691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:55.003268 17691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.003284 17691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:55.003307 17691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.008440 17691 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:36807
I20260620 19:20:55.008507 17821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:36807 every 8 connection(s)
I20260620 19:20:55.008770 17691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:20:55.009658 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.009341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:55117)
0620 19:20:55.009485 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.009539 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
W20260620 19:20:55.009989 17822 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44875 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44875: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
I20260620 19:20:55.010236 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.010157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:43665)
0620 19:20:55.010195 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.010217 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:20:55.010573 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17691
I20260620 19:20:55.010602 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.010506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:39389)
0620 19:20:55.010550 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.010578 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:20:55.010706 17822 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:20:55.010872 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:34161
--local_ip_for_outbound_sockets=127.1.1.195
--tserver_master_addrs=127.1.1.254:44875
--webserver_port=41343
--webserver_interface=127.1.1.195
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:20:55.082468 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:20:55.082625 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:20:55.082645 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:20:55.083719 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:20:55.083758 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:20:55.083782 17825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:20:55.085120 17825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:34161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=41343
--tserver_master_addrs=127.1.1.254:44875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:20:55.085335 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:20:55.085526 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:20:55.089705 17833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:20:55.089735 17831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:55.089818 17825 server_base.cc:1034] running on GCE node
W20260620 19:20:55.089926 17830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:20:55.090107 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:20:55.090287 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:20:55.091450 17825 hybrid_clock.cc:648] HybridClock initialized: now 1781983255091428 us; error 39 us; skew 500 ppm
Jun 20 19:20:55 dist-test-slave-mwkb krb5kdc[16390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:20:55.093657 17825 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:20:55.094039 17825 webserver.cc:460] Webserver started at http://127.1.1.195:41343/ using document root <none> and password file <none>
I20260620 19:20:55.094189 17825 fs_manager.cc:359] Metadata directory not provided
I20260620 19:20:55.094233 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:20:55.095336 17825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:20:55.095808 17840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.095988 17825 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.096073 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "60b91ad058a44424bf30fdee66f38d75"
format_stamp: "Formatted at 2026-06-20 19:20:13 on dist-test-slave-mwkb"
server_key: "d09398a655b0485497e6d680efaa1696"
server_key_iv: "f099a193efb3ba65a621e5abed900ab9"
server_key_version: "encryptionkey@0"
I20260620 19:20:55.096350 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:20:55.114127 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:20:55.115590 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:20:55.115720 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:20:55.115896 17825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:20:55.116137 17825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:20:55.116182 17825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.116216 17825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:20:55.116266 17825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:20:55.121520 17825 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34161
I20260620 19:20:55.121563 17953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34161 every 8 connection(s)
I20260620 19:20:55.121840 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:20:55.122686 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.122388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:39697)
0620 19:20:55.122524 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.122578 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
W20260620 19:20:55.123047 17954 heartbeater.cc:644] Failed to heartbeat to 127.1.1.254:44875 (0 consecutive failures): Network error: Failed to ping master at 127.1.1.254:44875: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
I20260620 19:20:55.123296 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.123198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:52787)
0620 19:20:55.123236 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.123262 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:20:55.123556 17956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.123490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:33371)
0620 19:20:55.123524 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.123543 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:20:55.123682 17954 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:20:55.125188 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 17825
I20260620 19:20:55.938273 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:55.938076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:55115)
0620 19:20:55.938186 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:55.938218 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260620 19:20:56.011154 17959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.010977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:38639)
0620 19:20:56.011084 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:56.011113 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260620 19:20:56.124099 17960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.123940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:59265)
0620 19:20:56.124034 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:56.124063 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":48,"threads_started":1}
I20260620 19:20:56.324621 17961 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:56.324733 17961 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:56.325126 17961 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
Jun 20 19:20:56 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.194@KRBTEST.COM
Jun 20 19:20:56 dist-test-slave-mwkb krb5kdc[16390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.195@KRBTEST.COM
I20260620 19:20:56.334492 17958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.325222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:36807 (local address 127.1.1.193:48137)
0620 19:20:56.325269 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:56.325280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:56.325335 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:56.326182 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:56.326185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:56.326534 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:56.326695 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:56.326700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.327666 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.327667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:56.328337 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:56.328339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.328844 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:56.328957 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:56.329576 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:56.329581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:56.330984 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:56.332983 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:56.332988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:56.332990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:56.333259 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:56.333479 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:56.333480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:56.333578 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:56.333988 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:56.334007 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:56.334134 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:20:56.334233 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:20:56.334392 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":57}
I20260620 19:20:56.334508 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.325344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:48137 (local address 127.1.1.194:36807)
0620 19:20:56.325495 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:56.325506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:56.325561 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:20:56.325978 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:56.325979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:56.326016 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:56.326121 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:56.326821 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.327558 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328422 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.328759 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328853 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:56.331089 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:56.331092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:56.331092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:56.331102 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:56.332912 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333328 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:56.333330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:56.333330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:56.333402 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333664 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:56.333665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:56.333665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:56.333922 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:56.334006 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:56.334298 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:20:56.334412 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 19:20:56.334508 17964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.325398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:43361 (local address 127.1.1.195:34161)
0620 19:20:56.325548 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:20:56.325557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:20:56.325587 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:20:56.326068 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:20:56.326070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:20:56.326133 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:20:56.326254 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:20:56.326808 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.327547 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328424 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.328762 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328876 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:56.331509 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:20:56.331511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:20:56.331512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:20:56.331531 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:20:56.333181 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333475 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:56.333477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:56.333477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:56.333572 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333745 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:20:56.333745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:20:56.333746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:20:56.333963 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:20:56.334028 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:20:56.334298 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 19:20:56.334412 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260620 19:20:56.334537 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.325234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34161 (local address 127.1.1.193:43361)
0620 19:20:56.325464 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:56.325474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:20:56.325516 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:20:56.326258 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:20:56.326259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:20:56.326559 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:20:56.326695 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:56.326700 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.327638 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.327640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:56.328336 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:20:56.328338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:20:56.328860 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:20:56.328861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:20:56.328962 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:20:56.329534 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:20:56.329541 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:20:56.331413 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:20:56.333243 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:56.333244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:56.333405 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:56.333638 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:20:56.333639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:20:56.333639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:20:56.333692 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:20:56.334021 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:20:56.334021 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:20:56.334134 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 19:20:56.334233 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 19:20:56.334439 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":109,"thread_start_us":52,"threads_started":1}
W20260620 19:20:56.335927 17577 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
W20260620 19:20:56.336092 17575 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:20:56.336138 17575 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b; no voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71
I20260620 19:20:56.336246 17961 raft_consensus.cc:2726] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:20:56.938930 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:56.938700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:35363)
0620 19:20:56.938851 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:56.938885 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260620 19:20:57.011744 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:57.011554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:43389)
0620 19:20:57.011677 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:57.011710 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260620 19:20:57.124874 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:57.124602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:46053)
0620 19:20:57.124814 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:57.124844 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":63,"threads_started":1}
I20260620 19:20:57.939605 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:57.939417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:59035)
0620 19:20:57.939533 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:57.939562 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260620 19:20:58.012279 17969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:58.012104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:57881)
0620 19:20:58.012221 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:58.012249 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260620 19:20:58.125422 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:58.125246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:46387)
0620 19:20:58.125361 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:58.125393 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":53,"threads_started":1}
I20260620 19:20:58.263674 17971 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:20:58.263756 17971 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:20:58.263914 17971 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
W20260620 19:20:58.264292 17577 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
W20260620 19:20:58.264381 17575 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:20:58.264413 17575 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b; no voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71
I20260620 19:20:58.264495 17971 raft_consensus.cc:2726] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:20:58.940248 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:58.940012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:43363)
0620 19:20:58.940162 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:58.940191 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":75,"threads_started":1}
I20260620 19:20:59.012769 17973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:59.012612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:60243)
0620 19:20:59.012710 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:59.012739 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
I20260620 19:20:59.125922 17974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:59.125753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:53781)
0620 19:20:59.125867 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:59.125895 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
I20260620 19:20:59.941021 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:20:59.940810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:56645)
0620 19:20:59.940943 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:20:59.940975 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260620 19:21:00.013412 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:00.013248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:40827)
0620 19:21:00.013353 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:00.013382 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260620 19:21:00.126447 17977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:00.126289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:60775)
0620 19:21:00.126389 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:00.126416 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260620 19:21:00.941617 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:00.941431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:35429)
0620 19:21:00.941541 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:00.941574 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260620 19:21:01.013979 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:01.013777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:52769)
0620 19:21:01.013917 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:01.013948 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260620 19:21:01.127045 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:01.126830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:40715)
0620 19:21:01.126963 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:01.126995 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
I20260620 19:21:01.219696 17981 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:01.219787 17981 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:21:01.219980 17981 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
W20260620 19:21:01.220300 17575 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
W20260620 19:21:01.220384 17577 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:21:01.220460 17577 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b; no voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71
I20260620 19:21:01.220523 17981 raft_consensus.cc:2726] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:21:01.942214 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:01.942011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:56127)
0620 19:21:01.942136 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:01.942169 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260620 19:21:02.014590 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:02.014435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:38495)
0620 19:21:02.014531 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:02.014560 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":50,"threads_started":1}
I20260620 19:21:02.127619 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:02.127455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:58199)
0620 19:21:02.127557 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:02.127584 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260620 19:21:02.942946 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:02.942770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:47057)
0620 19:21:02.942877 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:02.942906 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260620 19:21:03.015129 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:03.014954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:42881)
0620 19:21:03.015075 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:03.015101 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260620 19:21:03.128221 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:03.128029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:47541)
0620 19:21:03.128159 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:03.128189 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260620 19:21:03.943491 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:03.943328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:48529)
0620 19:21:03.943424 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:03.943450 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260620 19:21:04.015710 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:04.015541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:34131)
0620 19:21:04.015641 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:04.015680 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260620 19:21:04.128873 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:04.128697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:43083)
0620 19:21:04.128811 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:04.128837 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260620 19:21:04.944203 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:04.943856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:56917)
0620 19:21:04.944102 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:04.944147 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":72,"threads_started":1}
I20260620 19:21:05.016314 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:05.016140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:50523)
0620 19:21:05.016252 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:05.016281 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260620 19:21:05.129385 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:05.129222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:57311)
0620 19:21:05.129323 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:05.129354 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260620 19:21:05.711238 17994 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:05.711321 17994 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:21:05.711521 17994 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
W20260620 19:21:05.711850 17575 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
W20260620 19:21:05.711927 17577 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:21:05.712018 17577 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b; no voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71
I20260620 19:21:05.712116 17994 raft_consensus.cc:2726] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:21:05.944864 17995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:05.944660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:40823)
0620 19:21:05.944787 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:05.944820 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260620 19:21:06.017107 17996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:06.016842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:35143)
0620 19:21:06.016983 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:06.017017 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260620 19:21:06.129931 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:06.129746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:49479)
0620 19:21:06.129868 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:06.129898 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260620 19:21:06.945495 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:06.945279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:50885)
0620 19:21:06.945404 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:06.945453 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260620 19:21:07.017625 17999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:07.017479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:41557)
0620 19:21:07.017571 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:07.017597 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260620 19:21:07.130530 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:07.130278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:46227)
0620 19:21:07.130468 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:07.130498 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":49,"threads_started":1}
I20260620 19:21:07.946172 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:07.945947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:42067)
0620 19:21:07.946085 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:07.946131 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260620 19:21:08.018220 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:08.018039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:39407)
0620 19:21:08.018140 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:08.018187 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":49,"threads_started":1}
I20260620 19:21:08.131160 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:08.130978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:60447)
0620 19:21:08.131100 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:08.131129 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260620 19:21:08.946772 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:08.946598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:52615)
0620 19:21:08.946703 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:08.946731 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260620 19:21:09.018743 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:09.018580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:38495)
0620 19:21:09.018681 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:09.018709 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
I20260620 19:21:09.131672 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:09.131515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:36303)
0620 19:21:09.131614 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:09.131643 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260620 19:21:09.721237 18007 raft_consensus.cc:482] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:09.721329 18007 raft_consensus.cc:504] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b91ad058a44424bf30fdee66f38d75" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34161 } } peers { permanent_uuid: "af8862ccc3794783b68aedfea3485e71" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 36807 } } peers { permanent_uuid: "738c4c4efbc2490a9480395cc5ca784b" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 39053 } }
I20260620 19:21:09.721524 18007 leader_election.cc:288] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161), af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807)
W20260620 19:21:09.721912 17575 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 60b91ad058a44424bf30fdee66f38d75 (127.1.1.195:34161): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
W20260620 19:21:09.721982 17577 leader_election.cc:341] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer af8862ccc3794783b68aedfea3485e71 (127.1.1.194:36807): Not found: Tablet not found: c4ae5e8246164d04a0fd972d8ed31cd6
I20260620 19:21:09.722043 17577 leader_election.cc:302] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 738c4c4efbc2490a9480395cc5ca784b; no voters: 60b91ad058a44424bf30fdee66f38d75, af8862ccc3794783b68aedfea3485e71
I20260620 19:21:09.722249 18007 raft_consensus.cc:2726] T c4ae5e8246164d04a0fd972d8ed31cd6 P 738c4c4efbc2490a9480395cc5ca784b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:21:09.947422 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:09.947230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.193:37067)
0620 19:21:09.947338 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:09.947370 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260620 19:21:10.019358 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:10.019157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.194:40587)
0620 19:21:10.019290 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:10.019324 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260620 19:21:10.132320 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:10.132123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:44875 (local address 127.1.1.195:57193)
0620 19:21:10.132255 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:10.132285 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.254:44875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260620 19:21:10.134229 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17527
I20260620 19:21:10.138696 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17691
I20260620 19:21:10.142544 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 17825
2026-06-20T19:21:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (57907 ms)
[----------] 30 tests from SecurityITest (248489 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:10 dist-test-slave-mwkb krb5kdc[18013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:10 dist-test-slave-mwkb krb5kdc[18013](info): set up 2 sockets
Jun 20 19:21:10 dist-test-slave-mwkb krb5kdc[18013](info): commencing operation
krb5kdc: starting...
W20260620 19:21:11.088122 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:11.185418 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41675
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:11.254122 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:11.254305 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:11.254338 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:11.255293 18026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:11.255352 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:11.255374 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:11.255398 18026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:11.255422 18026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:11.256629 18026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:11.256860 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:11.257043 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:11.260970 18034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:11.260979 18032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.261165 18026 server_base.cc:1034] running on GCE node
W20260620 19:21:11.261109 18031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.261435 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:11.261658 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:11.262794 18026 hybrid_clock.cc:648] HybridClock initialized: now 1781983271262784 us; error 27 us; skew 500 ppm
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:11.264765 18026 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:11.265172 18026 webserver.cc:460] Webserver started at http://127.1.1.254:36589/ using document root <none> and password file <none>
I20260620 19:21:11.265349 18026 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:11.265385 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:11.265470 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:11.266276 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "37c3201bf06642b5beb9099d5ae5edda"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "598c077a297c8f26b926167dd6394b87"
server_key_iv: "994c4783fea37d73105f521347fd37ba"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.266535 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "37c3201bf06642b5beb9099d5ae5edda"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "598c077a297c8f26b926167dd6394b87"
server_key_iv: "994c4783fea37d73105f521347fd37ba"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.267583 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:21:11.268139 18041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.268293 18026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:11.268386 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "37c3201bf06642b5beb9099d5ae5edda"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "598c077a297c8f26b926167dd6394b87"
server_key_iv: "994c4783fea37d73105f521347fd37ba"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.268435 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:11.280970 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:11.282655 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:11.282765 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:11.285966 18026 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:41675
I20260620 19:21:11.286026 18093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:41675 every 8 connection(s)
I20260620 19:21:11.286266 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:11.286962 18094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:11.288690 18094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: Bootstrap starting.
I20260620 19:21:11.288830 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18026
I20260620 19:21:11.288928 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:11.289126 1031 external_mini_cluster.cc:1428] Setting key 73a62d500356a50c930c3c57fc1361ad
I20260620 19:21:11.289372 18094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:11.289671 18094 log.cc:824] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:11.290298 18094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: No bootstrap required, opened a new log
I20260620 19:21:11.292281 18094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } }
I20260620 19:21:11.292441 18094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:11.292467 18094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c3201bf06642b5beb9099d5ae5edda, State: Initialized, Role: FOLLOWER
I20260620 19:21:11.292613 18094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } }
I20260620 19:21:11.292686 18094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:11.292707 18094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:11.292781 18094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:11.293408 18094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } }
I20260620 19:21:11.293497 18094 leader_election.cc:302] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37c3201bf06642b5beb9099d5ae5edda; no voters:
I20260620 19:21:11.293629 18094 leader_election.cc:288] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:11.293689 18099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:11.293830 18094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:11.293848 18099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [term 1 LEADER]: Becoming Leader. State: Replica: 37c3201bf06642b5beb9099d5ae5edda, State: Running, Role: LEADER
I20260620 19:21:11.293922 18099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } }
I20260620 19:21:11.294212 18099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c3201bf06642b5beb9099d5ae5edda. Latest consensus state: current_term: 1 leader_uuid: "37c3201bf06642b5beb9099d5ae5edda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } } }
I20260620 19:21:11.294430 18099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:11.294368 18100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37c3201bf06642b5beb9099d5ae5edda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37c3201bf06642b5beb9099d5ae5edda" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41675 } } }
I20260620 19:21:11.294764 18100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:11.294879 18108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:11.295547 18108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:11.298450 18108 catalog_manager.cc:1347] Generated new cluster ID: d11ad560ba704434ac9cc291a1f04f97
I20260620 19:21:11.298506 18108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:11.299839 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.289543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.1.1.254:41675)
0620 19:21:11.289784 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.289793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.289822 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:11.290290 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.290297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.290347 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.290473 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:11.290961 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.292261 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.293315 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.293866 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.295261 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.296493 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:11.296496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:11.296497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:11.296506 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:11.298240 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.298656 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.298659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.298660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.298760 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.298980 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.298981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.298982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.299305 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:11.299453 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.299520 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.299648 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":51,"threads_started":1}
I20260620 19:21:11.311139 18108 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:11.311558 18108 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:11.313490 18108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: Generated new TSK 0
I20260620 19:21:11.313619 18108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:21:11.340029 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:11.432271 18120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:11.432448 18120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:11.432471 18120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:11.433578 18120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:11.433614 18120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:11.433636 18120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:11.434937 18120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:11.435120 18120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:11.435331 18120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:11.439589 18126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.439664 18120 server_base.cc:1034] running on GCE node
W20260620 19:21:11.439836 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:11.439884 18125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.440092 18120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:11.440318 18120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:11.441469 18120 hybrid_clock.cc:648] HybridClock initialized: now 1781983271441463 us; error 31 us; skew 500 ppm
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:11.443686 18120 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:21:11.444108 18120 webserver.cc:460] Webserver started at http://127.1.1.193:41081/ using document root <none> and password file <none>
I20260620 19:21:11.444296 18120 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:11.444352 18120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:11.444463 18120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:11.445348 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "84c1357cf82947e1a4d5e56df55d929d"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "b545ed2f0f0d35e4bc2a6e71a6b0b590"
server_key_iv: "836abd0c76aecaea671a37edfea78b48"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.445658 18120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "84c1357cf82947e1a4d5e56df55d929d"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "b545ed2f0f0d35e4bc2a6e71a6b0b590"
server_key_iv: "836abd0c76aecaea671a37edfea78b48"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.446817 18120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:11.447458 18135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.447614 18120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:11.447697 18120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "84c1357cf82947e1a4d5e56df55d929d"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "b545ed2f0f0d35e4bc2a6e71a6b0b590"
server_key_iv: "836abd0c76aecaea671a37edfea78b48"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.447769 18120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:11.454469 18120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:11.455874 18120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:11.456002 18120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:11.456172 18120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:11.456415 18120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:11.456461 18120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.456494 18120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:11.456516 18120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.462003 18120 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:41669
I20260620 19:21:11.462069 18248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:41669 every 8 connection(s)
I20260620 19:21:11.462332 18120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:21:11.464175 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18120
I20260620 19:21:11.464277 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:21:11.464438 1031 external_mini_cluster.cc:1428] Setting key 9f6fc70525271fce9600445b8c9a9fba
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:11.471524 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.462858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:57449 (local address 127.1.1.254:41675)
0620 19:21:11.462902 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.462904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.463152 (+ 248us) server_negotiation.cc:407] Connection header received
0620 19:21:11.463514 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.463515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.463523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.463567 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:11.464329 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.464926 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.466209 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.466497 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.466560 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.468720 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:11.468721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:11.468721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:11.468727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:11.470282 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.470633 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.470634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.470635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.470693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.470944 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.470945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.470945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.471069 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:11.471136 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.471355 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.471431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:21:11.471532 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.462862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41675 (local address 127.1.1.193:57449)
0620 19:21:11.463038 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.463068 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.463450 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.463621 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.463623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.464082 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:11.464229 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.464232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.465025 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.465026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.466128 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.466131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.466580 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.466581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.466703 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.467078 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:11.467086 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:11.468636 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:11.470363 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.470368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.470370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.470562 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.470757 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.470757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.470758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.470838 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.471142 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:11.471146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:11.471240 (+ 94us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.471308 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.471404 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260620 19:21:11.472054 18249 heartbeater.cc:347] Connected to a master server at 127.1.1.254:41675
I20260620 19:21:11.472144 18249 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:11.472306 18249 heartbeater.cc:511] Master 127.1.1.254:41675 requested a full tablet report, sending...
I20260620 19:21:11.472826 18058 ts_manager.cc:194] Registered new tserver with Master: 84c1357cf82947e1a4d5e56df55d929d (127.1.1.193:41669)
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
I20260620 19:21:11.473551 18058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:57449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:21:11.485759 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:11.556187 18254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:11.556340 18254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:11.556360 18254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:11.557323 18254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:11.557356 18254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:11.557379 18254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:11.558509 18254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:11.558647 18254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:11.558826 18254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:11.563110 18262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:11.563124 18260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:11.563128 18259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.563407 18254 server_base.cc:1034] running on GCE node
I20260620 19:21:11.563572 18254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:11.563727 18254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:11.564850 18254 hybrid_clock.cc:648] HybridClock initialized: now 1781983271564839 us; error 24 us; skew 500 ppm
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:11.566843 18254 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:21:11.567198 18254 webserver.cc:460] Webserver started at http://127.1.1.194:45525/ using document root <none> and password file <none>
I20260620 19:21:11.567376 18254 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:11.567412 18254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:11.567539 18254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:11.568380 18254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "345655d1ea594d7aa8c3358bba74962f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "1ed26f2f588a091338f8831e99bf53e3"
server_key_iv: "bcc2deadd34fdac8b12a6ae8eb56151c"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.568719 18254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "345655d1ea594d7aa8c3358bba74962f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "1ed26f2f588a091338f8831e99bf53e3"
server_key_iv: "bcc2deadd34fdac8b12a6ae8eb56151c"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.570036 18254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:11.570724 18269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.570897 18254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:11.570989 18254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "345655d1ea594d7aa8c3358bba74962f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "1ed26f2f588a091338f8831e99bf53e3"
server_key_iv: "bcc2deadd34fdac8b12a6ae8eb56151c"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.571065 18254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:11.589825 18254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:11.591334 18254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:11.591477 18254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:11.591658 18254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:11.591900 18254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:11.591954 18254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.591989 18254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:11.592031 18254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.597406 18254 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:40713
I20260620 19:21:11.597472 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:40713 every 8 connection(s)
I20260620 19:21:11.597733 18254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:21:11.599385 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18254
I20260620 19:21:11.599457 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:11.599627 1031 external_mini_cluster.cc:1428] Setting key 34f8450572a0233912d2a934b39579c9
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:11.606544 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.598289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:50301 (local address 127.1.1.254:41675)
0620 19:21:11.598326 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.598327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.598494 (+ 167us) server_negotiation.cc:407] Connection header received
0620 19:21:11.598838 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.598839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.598847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.598889 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:11.599752 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.600428 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.601348 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.601552 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.601642 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.603846 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:11.603847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:11.603848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:11.603853 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:11.605275 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.605752 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.605753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.605753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.605795 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.606027 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.606028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.606028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.606111 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:11.606153 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.606439 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.606491 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:21:11.606763 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.598280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41675 (local address 127.1.1.194:50301)
0620 19:21:11.598397 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.598423 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.598774 (+ 351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.598930 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.598932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.599411 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:11.599647 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.599674 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.600521 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.600522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.601269 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.601271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.601646 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.601647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.601773 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.602228 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:11.602235 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:11.603766 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:11.605354 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.605361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.605362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.605672 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.605854 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.605855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.605855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.605969 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.606169 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:11.606174 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:11.606317 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.606423 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.606561 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
I20260620 19:21:11.607204 18383 heartbeater.cc:347] Connected to a master server at 127.1.1.254:41675
I20260620 19:21:11.607293 18383 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:11.607440 18383 heartbeater.cc:511] Master 127.1.1.254:41675 requested a full tablet report, sending...
I20260620 19:21:11.607738 18058 ts_manager.cc:194] Registered new tserver with Master: 345655d1ea594d7aa8c3358bba74962f (127.1.1.194:40713)
I20260620 19:21:11.608076 18058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:50301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:21:11.621423 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:37267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:11.691215 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:11.691393 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:11.691426 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:11.692512 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:11.692577 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:11.692621 18388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:11.694175 18388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:37267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:11.694410 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:11.694641 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:11.699028 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.699132 18388 server_base.cc:1034] running on GCE node
W20260620 19:21:11.699015 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:11.699018 18396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:11.699383 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:11.699602 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:11.700765 18388 hybrid_clock.cc:648] HybridClock initialized: now 1781983271700760 us; error 33 us; skew 500 ppm
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:11.703015 18388 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:21:11.703418 18388 webserver.cc:460] Webserver started at http://127.1.1.195:36667/ using document root <none> and password file <none>
I20260620 19:21:11.703619 18388 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:11.703680 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:11.703814 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:11.704638 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "35566a9c45474c34b7c343dbb829fe5f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "636469a4a38c3d8faa56354777d30c12"
server_key_iv: "89a04a3d94c08c3445c5c39eb5c1af02"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.704981 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "35566a9c45474c34b7c343dbb829fe5f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "636469a4a38c3d8faa56354777d30c12"
server_key_iv: "89a04a3d94c08c3445c5c39eb5c1af02"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.706130 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:21:11.706729 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.706888 18388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:11.706974 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "35566a9c45474c34b7c343dbb829fe5f"
format_stamp: "Formatted at 2026-06-20 19:21:11 on dist-test-slave-mwkb"
server_key: "636469a4a38c3d8faa56354777d30c12"
server_key_iv: "89a04a3d94c08c3445c5c39eb5c1af02"
server_key_version: "encryptionkey@0"
I20260620 19:21:11.707057 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:11.715643 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:11.717408 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:11.717545 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:11.717720 18388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:11.717958 18388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:11.718004 18388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.718039 18388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:11.718062 18388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:11.723037 18388 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:35791
I20260620 19:21:11.723099 18516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:35791 every 8 connection(s)
I20260620 19:21:11.723373 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:21:11.725041 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18388
I20260620 19:21:11.725152 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:11.725307 1031 external_mini_cluster.cc:1428] Setting key 494e438e89a617a5807c1f6d5df92638
Jun 20 19:21:11 dist-test-slave-mwkb krb5kdc[18013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983271, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:11.730930 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.723953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:42511 (local address 127.1.1.254:41675)
0620 19:21:11.724003 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.724004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.724166 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:21:11.724551 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.724552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.724563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.724607 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:11.725444 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.726062 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.726880 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.727063 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.727135 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.728690 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:11.728691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:11.728691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:11.728697 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:11.729731 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.730120 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.730121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.730122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.730173 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.730397 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.730398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.730398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.730496 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:11.730562 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.730789 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.730855 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:21:11.731058 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.723936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41675 (local address 127.1.1.195:42511)
0620 19:21:11.724060 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.724109 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.724463 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.724668 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.724671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.725190 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:11.725375 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.725380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.726141 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.726142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.726791 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.726793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.727144 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.727145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.727252 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.727563 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:11.727570 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:11.728607 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:11.729817 (+ 1210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.729823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.729824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.730004 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.730238 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:11.730239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:11.730239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:11.730335 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:11.730554 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:11.730559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:11.730673 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.730764 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.730883 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
I20260620 19:21:11.731441 18517 heartbeater.cc:347] Connected to a master server at 127.1.1.254:41675
I20260620 19:21:11.731500 18517 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:11.731652 18517 heartbeater.cc:511] Master 127.1.1.254:41675 requested a full tablet report, sending...
I20260620 19:21:11.731947 18058 ts_manager.cc:194] Registered new tserver with Master: 35566a9c45474c34b7c343dbb829fe5f (127.1.1.195:35791)
I20260620 19:21:11.732270 18058 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:42511
I20260620 19:21:11.736812 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:11.743472 18097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.737933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.1.1.254:41675)
0620 19:21:11.738006 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.738007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.738102 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:21:11.738182 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.738182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.738190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.738224 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:11.738702 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.739088 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.739801 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.739993 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.740062 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.741107 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:11.741108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:11.741108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:11.741114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:11.742321 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.742685 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.742686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.742686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.742737 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:11.742937 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:11.742938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:11.742938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:11.743077 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:11.743180 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.743326 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.743388 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:21:11.744659 18058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:11.746135 18058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:11.751432 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.747983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35791 (local address 127.0.0.1:50054)
0620 19:21:11.748414 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.748423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.748475 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.748641 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.748642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.748643 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:11.748679 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.748681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749537 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.749538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.750452 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.750454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751178 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.751248 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.751271 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.751333 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.751376 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260620 19:21:11.751441 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.748043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.1.1.195:35791)
0620 19:21:11.748151 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.748155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.748525 (+ 370us) server_negotiation.cc:407] Connection header received
0620 19:21:11.748554 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.748555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.748566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.748644 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:11.748724 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749449 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.750521 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751086 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751176 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.751280 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.751343 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.751372 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":58,"threads_started":1}
I20260620 19:21:11.751786 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.747795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:41669 (local address 127.0.0.1:54116)
0620 19:21:11.747959 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.747993 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.748047 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.748340 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.748342 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.748344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:11.748395 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.748397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749581 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.749581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.750670 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.750672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751575 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.751627 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.751630 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.751677 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.751708 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":79,"thread_start_us":54,"threads_started":1}
I20260620 19:21:11.751816 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.747908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.1.1.193:41669)
0620 19:21:11.748194 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.748198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.748206 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:11.748244 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.748246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.748257 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.748314 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:11.748471 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749476 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.750773 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751427 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751489 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.751540 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.751709 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.751749 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":64,"threads_started":1}
I20260620 19:21:11.752017 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea04337a8654facac9ceab91bf5f85e (DEFAULT_TABLE table=test-table [id=83c21340c88441bfa2d655c293e1e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:11.752216 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.747828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:40713 (local address 127.0.0.1:60166)
0620 19:21:11.748140 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:11.748147 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:11.748178 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:11.748400 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:11.748401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:11.748403 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:11.748451 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.748453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749580 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.749581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.750676 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:11.750678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751792 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:11.752016 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.752020 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:11.752086 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:21:11.752129 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":89,"thread_start_us":67,"threads_started":1}
I20260620 19:21:11.752341 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea04337a8654facac9ceab91bf5f85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:11.752457 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:11.748063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60166 (local address 127.1.1.194:40713)
0620 19:21:11.748229 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:11.748232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:11.748240 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:11.748283 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:11.748284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:11.748311 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:11.748384 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:11.748517 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.749485 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.750767 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:11.751429 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:11.751702 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:11.751760 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:11.752335 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 19:21:11.752380 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260620 19:21:11.753108 18317 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea04337a8654facac9ceab91bf5f85e (DEFAULT_TABLE table=test-table [id=83c21340c88441bfa2d655c293e1e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:11.753449 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea04337a8654facac9ceab91bf5f85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:11.754856 18531 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Bootstrap starting.
I20260620 19:21:11.755281 18183 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea04337a8654facac9ceab91bf5f85e (DEFAULT_TABLE table=test-table [id=83c21340c88441bfa2d655c293e1e1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:11.755517 18531 tablet_bootstrap.cc:654] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:11.755609 18183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea04337a8654facac9ceab91bf5f85e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:11.756016 18531 log.cc:824] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:11.756176 18532 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Bootstrap starting.
I20260620 19:21:11.756764 18531 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: No bootstrap required, opened a new log
I20260620 19:21:11.756798 18532 tablet_bootstrap.cc:654] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:11.756878 18531 ts_tablet_manager.cc:1397] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:11.757296 18532 log.cc:824] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:11.757953 18532 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: No bootstrap required, opened a new log
I20260620 19:21:11.758018 18532 ts_tablet_manager.cc:1397] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:21:11.758283 18535 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Bootstrap starting.
I20260620 19:21:11.758767 18535 tablet_bootstrap.cc:654] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:11.758904 18531 raft_consensus.cc:348] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.759057 18531 raft_consensus.cc:374] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:11.759093 18531 raft_consensus.cc:729] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35566a9c45474c34b7c343dbb829fe5f, State: Initialized, Role: FOLLOWER
I20260620 19:21:11.759227 18531 consensus_queue.cc:260] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.759503 18531 ts_tablet_manager.cc:1428] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:21:11.759611 18517 heartbeater.cc:503] Master 127.1.1.254:41675 was elected leader, sending a full tablet report...
I20260620 19:21:11.759596 18535 log.cc:824] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:11.759865 18532 raft_consensus.cc:348] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.760044 18532 raft_consensus.cc:374] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:11.760090 18532 raft_consensus.cc:729] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345655d1ea594d7aa8c3358bba74962f, State: Initialized, Role: FOLLOWER
I20260620 19:21:11.760216 18532 consensus_queue.cc:260] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.760308 18535 tablet_bootstrap.cc:492] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: No bootstrap required, opened a new log
I20260620 19:21:11.760488 18532 ts_tablet_manager.cc:1428] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:21:11.760496 18383 heartbeater.cc:503] Master 127.1.1.254:41675 was elected leader, sending a full tablet report...
I20260620 19:21:11.760592 18535 ts_tablet_manager.cc:1397] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:11.762316 18535 raft_consensus.cc:348] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.762499 18535 raft_consensus.cc:374] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:11.762547 18535 raft_consensus.cc:729] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84c1357cf82947e1a4d5e56df55d929d, State: Initialized, Role: FOLLOWER
I20260620 19:21:11.762653 18535 consensus_queue.cc:260] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:11.762938 18535 ts_tablet_manager.cc:1428] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:11.763273 18249 heartbeater.cc:503] Master 127.1.1.254:41675 was elected leader, sending a full tablet report...
W20260620 19:21:11.848204 18384 tablet.cc:2369] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:21:11.962940 18250 tablet.cc:2369] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:21:11.973846 18518 tablet.cc:2369] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:12.125190 18538 raft_consensus.cc:482] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:12.125290 18538 raft_consensus.cc:504] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:12.125623 18538 leader_election.cc:288] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84c1357cf82947e1a4d5e56df55d929d (127.1.1.193:41669), 35566a9c45474c34b7c343dbb829fe5f (127.1.1.195:35791)
I20260620 19:21:12.128319 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.126076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:56011 (local address 127.1.1.195:35791)
0620 19:21:12.126151 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.126152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.126178 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:21:12.126198 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.126199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.126206 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.126240 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.126396 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.126811 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.127692 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.128013 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.128087 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.128096 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.128221 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.128254 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:21:12.128333 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.125897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:47171 (local address 127.1.1.193:41669)
0620 19:21:12.125965 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.125967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.126017 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:21:12.126127 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.126128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.126141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.126168 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.126427 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.126883 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.127719 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.128016 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.128090 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.128099 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.128245 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.128277 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:21:12.128341 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.125806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:41669 (local address 127.1.1.194:47171)
0620 19:21:12.125953 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.125964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.126032 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.126250 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.126251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.126252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.126333 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.126335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.126963 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.126963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.127619 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.127620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.128094 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.128095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.128159 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.128164 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.128225 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.128257 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":14,"thread_start_us":80,"threads_started":1}
I20260620 19:21:12.128343 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.126006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35791 (local address 127.1.1.194:56011)
0620 19:21:12.126077 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.126097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.126133 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.126274 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.126274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.126275 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.126334 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.126336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.126889 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.126891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.127610 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.127612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.128090 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.128091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.128157 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.128160 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.128196 (+ 36us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.128254 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":17,"thread_start_us":36,"threads_started":1}
I20260620 19:21:12.128648 18203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea04337a8654facac9ceab91bf5f85e" candidate_uuid: "345655d1ea594d7aa8c3358bba74962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c1357cf82947e1a4d5e56df55d929d" is_pre_election: true
I20260620 19:21:12.128780 18203 raft_consensus.cc:2454] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345655d1ea594d7aa8c3358bba74962f in term 0.
I20260620 19:21:12.128904 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea04337a8654facac9ceab91bf5f85e" candidate_uuid: "345655d1ea594d7aa8c3358bba74962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35566a9c45474c34b7c343dbb829fe5f" is_pre_election: true
I20260620 19:21:12.128998 18271 leader_election.cc:302] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345655d1ea594d7aa8c3358bba74962f, 84c1357cf82947e1a4d5e56df55d929d; no voters:
I20260620 19:21:12.129040 18471 raft_consensus.cc:2454] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345655d1ea594d7aa8c3358bba74962f in term 0.
I20260620 19:21:12.129158 18538 raft_consensus.cc:2781] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:12.129230 18538 raft_consensus.cc:482] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:12.129251 18538 raft_consensus.cc:3037] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.129853 18538 raft_consensus.cc:504] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:12.129958 18538 leader_election.cc:288] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [CANDIDATE]: Term 1 election: Requested vote from peers 84c1357cf82947e1a4d5e56df55d929d (127.1.1.193:41669), 35566a9c45474c34b7c343dbb829fe5f (127.1.1.195:35791)
I20260620 19:21:12.130098 18203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea04337a8654facac9ceab91bf5f85e" candidate_uuid: "345655d1ea594d7aa8c3358bba74962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84c1357cf82947e1a4d5e56df55d929d"
I20260620 19:21:12.130105 18471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea04337a8654facac9ceab91bf5f85e" candidate_uuid: "345655d1ea594d7aa8c3358bba74962f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35566a9c45474c34b7c343dbb829fe5f"
I20260620 19:21:12.130169 18471 raft_consensus.cc:3037] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.130175 18203 raft_consensus.cc:3037] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.130781 18203 raft_consensus.cc:2454] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345655d1ea594d7aa8c3358bba74962f in term 1.
I20260620 19:21:12.130788 18471 raft_consensus.cc:2454] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345655d1ea594d7aa8c3358bba74962f in term 1.
I20260620 19:21:12.130961 18271 leader_election.cc:302] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345655d1ea594d7aa8c3358bba74962f, 84c1357cf82947e1a4d5e56df55d929d; no voters:
I20260620 19:21:12.131038 18538 raft_consensus.cc:2781] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:12.131156 18538 raft_consensus.cc:686] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 1 LEADER]: Becoming Leader. State: Replica: 345655d1ea594d7aa8c3358bba74962f, State: Running, Role: LEADER
I20260620 19:21:12.131234 18538 consensus_queue.cc:237] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } }
I20260620 19:21:12.131911 18057 catalog_manager.cc:5495] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f reported cstate change: term changed from 0 to 1, leader changed from <none> to 345655d1ea594d7aa8c3358bba74962f (127.1.1.194). New cstate: current_term: 1 leader_uuid: "345655d1ea594d7aa8c3358bba74962f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345655d1ea594d7aa8c3358bba74962f" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 40713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 } health_report { overall_health: UNKNOWN } } }
I20260620 19:21:12.187923 18530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.186086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.1.1.194:40713)
0620 19:21:12.186143 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.186145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.186151 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:21:12.186174 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.186175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.186188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.186238 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:12.186349 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.186718 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.187346 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.187505 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.187564 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.187690 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:12.187776 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:12.187839 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.187865 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.187879 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:21:12.189448 18471 raft_consensus.cc:1261] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 1 FOLLOWER]: Refusing update from remote peer 345655d1ea594d7aa8c3358bba74962f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:12.189455 18203 raft_consensus.cc:1261] T 7ea04337a8654facac9ceab91bf5f85e P 84c1357cf82947e1a4d5e56df55d929d [term 1 FOLLOWER]: Refusing update from remote peer 345655d1ea594d7aa8c3358bba74962f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:12.189751 18538 consensus_queue.cc:1035] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84c1357cf82947e1a4d5e56df55d929d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 41669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:12.189903 18542 consensus_queue.cc:1035] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [LEADER]: Connected to new peer: Peer: permanent_uuid: "35566a9c45474c34b7c343dbb829fe5f" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:12.192432 18546 mvcc.cc:205] Tried to move back new op lower bound from 7299003482885566464 to 7299003482649989120. Current Snapshot: MvccSnapshot[applied={T|T < 7299003482885566464}]
I20260620 19:21:12.193889 18058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:12.193976 18058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:12.194581 18058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c3201bf06642b5beb9099d5ae5edda: Sending DeleteTablet for 3 replicas of tablet 7ea04337a8654facac9ceab91bf5f85e
I20260620 19:21:12.194855 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea04337a8654facac9ceab91bf5f85e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:60166
I20260620 19:21:12.194907 18183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea04337a8654facac9ceab91bf5f85e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:54116
I20260620 19:21:12.194950 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ea04337a8654facac9ceab91bf5f85e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50054
I20260620 19:21:12.195164 18557 tablet_replica.cc:330] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: stopping tablet replica
I20260620 19:21:12.195246 18557 raft_consensus.cc:2229] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:12.195328 18555 tablet_replica.cc:330] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: stopping tablet replica
I20260620 19:21:12.195401 18555 raft_consensus.cc:2229] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:21:12.195429 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18120
I20260620 19:21:12.195468 18555 raft_consensus.cc:2258] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:12.196342 18555 ts_tablet_manager.cc:1905] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:12.197777 18555 ts_tablet_manager.cc:1918] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:12.197862 18555 log.cc:1196] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/7ea04337a8654facac9ceab91bf5f85e
I20260620 19:21:12.198119 18555 ts_tablet_manager.cc:1939] T 7ea04337a8654facac9ceab91bf5f85e P 345655d1ea594d7aa8c3358bba74962f: Deleting consensus metadata
I20260620 19:21:12.198483 18042 catalog_manager.cc:4841] TS 345655d1ea594d7aa8c3358bba74962f (127.1.1.194:40713): tablet 7ea04337a8654facac9ceab91bf5f85e (table test-table [id=83c21340c88441bfa2d655c293e1e1e1]) successfully deleted
W20260620 19:21:12.200915 18043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:41669 (error 108)
I20260620 19:21:12.201153 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18254
I20260620 19:21:12.201205 18557 raft_consensus.cc:2258] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:12.202112 18557 ts_tablet_manager.cc:1905] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:12.202447 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.202315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:41669 (local address 127.0.0.1:54124)
0620 19:21:12.202376 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.202417 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:41669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 19:21:12.202600 18043 catalog_manager.cc:4584] TS 84c1357cf82947e1a4d5e56df55d929d (127.1.1.193:41669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ea04337a8654facac9ceab91bf5f85e: Network error: Client connection negotiation failed: client connection to 127.1.1.193:41669: connect: Connection refused (error 111)
I20260620 19:21:12.203482 18557 ts_tablet_manager.cc:1918] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:12.203532 18557 log.cc:1196] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/7ea04337a8654facac9ceab91bf5f85e
I20260620 19:21:12.203759 18557 ts_tablet_manager.cc:1939] T 7ea04337a8654facac9ceab91bf5f85e P 35566a9c45474c34b7c343dbb829fe5f: Deleting consensus metadata
I20260620 19:21:12.204113 18043 catalog_manager.cc:4841] TS 35566a9c45474c34b7c343dbb829fe5f (127.1.1.195:35791): tablet 7ea04337a8654facac9ceab91bf5f85e (table test-table [id=83c21340c88441bfa2d655c293e1e1e1]) successfully deleted
I20260620 19:21:12.206190 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18388
I20260620 19:21:12.210387 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18026
2026-06-20T19:21:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2067 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:12Z Disabled control of system clock
I20260620 19:21:12.235494 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34677
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:34677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:12.306885 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:12.307111 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:12.307130 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:12.308141 18562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:12.308172 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:12.308185 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:12.308197 18562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:12.308220 18562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:12.309523 18562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:34677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:34677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:12.309688 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:12.309894 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:12.314116 18567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.314088 18568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.314091 18570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:12.314322 18562 server_base.cc:1034] running on GCE node
I20260620 19:21:12.314594 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:12.314826 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:12.315959 18562 hybrid_clock.cc:648] HybridClock initialized: now 1781983272315935 us; error 38 us; skew 500 ppm
I20260620 19:21:12.317036 18562 webserver.cc:460] Webserver started at http://127.1.1.254:36283/ using document root <none> and password file <none>
I20260620 19:21:12.317287 18562 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:12.317332 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:12.317472 18562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:12.318480 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "4e2705b1aa7843929a1a1cd9c2448307"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "9138d1776963853c5f50a117afa5f8d5"
server_key_iv: "42a506eb9f5780c96f52bb142d814ee7"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.318835 18562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "4e2705b1aa7843929a1a1cd9c2448307"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "9138d1776963853c5f50a117afa5f8d5"
server_key_iv: "42a506eb9f5780c96f52bb142d814ee7"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.320072 18562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:12.320714 18576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.320907 18562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:12.321019 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "4e2705b1aa7843929a1a1cd9c2448307"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "9138d1776963853c5f50a117afa5f8d5"
server_key_iv: "42a506eb9f5780c96f52bb142d814ee7"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.321138 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:12.334688 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:12.334914 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:12.335044 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:12.338385 18562 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:34677
I20260620 19:21:12.338414 18628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:34677 every 8 connection(s)
I20260620 19:21:12.338696 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:12.339107 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18562
I20260620 19:21:12.339206 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:12.339426 1031 external_mini_cluster.cc:1428] Setting key bb12fb5d4349af16757a8b3d858fd2ff
I20260620 19:21:12.339803 18629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:12.342533 18629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: Bootstrap starting.
I20260620 19:21:12.343130 18629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:12.343418 18629 log.cc:824] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:12.344168 18629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: No bootstrap required, opened a new log
I20260620 19:21:12.344799 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.339910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.1.1.254:34677)
0620 19:21:12.340109 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.340117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.340146 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:12.340741 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.340747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.340796 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.340888 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:12.341135 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.341872 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.343004 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.343443 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.343562 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.344189 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:12.344194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:12.344194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:12.344372 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:12.344406 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:12.344418 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:12.344492 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.344554 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.344622 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260620 19:21:12.346047 18629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } }
I20260620 19:21:12.346158 18629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:12.346179 18629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2705b1aa7843929a1a1cd9c2448307, State: Initialized, Role: FOLLOWER
I20260620 19:21:12.346248 18629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } }
I20260620 19:21:12.346292 18629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:12.346310 18629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:12.346333 18629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.346743 18629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } }
I20260620 19:21:12.346815 18629 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2705b1aa7843929a1a1cd9c2448307; no voters:
I20260620 19:21:12.346920 18629 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:12.346975 18634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:12.347096 18634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2705b1aa7843929a1a1cd9c2448307, State: Running, Role: LEADER
I20260620 19:21:12.347142 18629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:12.347204 18634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } }
I20260620 19:21:12.347628 18635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2705b1aa7843929a1a1cd9c2448307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } } }
I20260620 19:21:12.347733 18635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:12.347630 18636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2705b1aa7843929a1a1cd9c2448307. Latest consensus state: current_term: 1 leader_uuid: "4e2705b1aa7843929a1a1cd9c2448307" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2705b1aa7843929a1a1cd9c2448307" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 34677 } } }
I20260620 19:21:12.347851 18636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:12.348101 18643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:12.348456 18643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:12.349771 18643 catalog_manager.cc:1347] Generated new cluster ID: aaaeafe7db824d3eac602b478232f39a
I20260620 19:21:12.349820 18643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:12.360314 18643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:12.360740 18643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:12.366935 18643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: Generated new TSK 0
I20260620 19:21:12.367070 18643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:12.369566 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--builtin_ntp_servers=127.1.1.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:12.440122 18653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:12.440299 18653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:12.440318 18653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:12.441349 18653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:12.441380 18653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:12.441401 18653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:12.442608 18653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:12.442832 18653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:12.443110 18653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:21:12.448422 18653 server_base.cc:1034] running on GCE node
W20260620 19:21:12.448521 18658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.448397 18659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.448575 18661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:12.448860 18653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:12.449062 18653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:12.450237 18653 hybrid_clock.cc:648] HybridClock initialized: now 1781983272450228 us; error 23 us; skew 500 ppm
I20260620 19:21:12.451351 18653 webserver.cc:460] Webserver started at http://127.1.1.193:35571/ using document root <none> and password file <none>
I20260620 19:21:12.451570 18653 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:12.451615 18653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:12.451718 18653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:12.452989 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "1065c295b6b04d60808739ece561803d"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "2accfd43b3b2b8b13537666d0775c9b1"
server_key_iv: "1ee129896df7a2227798fd7bd267be86"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.453385 18653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "1065c295b6b04d60808739ece561803d"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "2accfd43b3b2b8b13537666d0775c9b1"
server_key_iv: "1ee129896df7a2227798fd7bd267be86"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.454663 18653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:12.455368 18667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.455554 18653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:12.455632 18653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "1065c295b6b04d60808739ece561803d"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "2accfd43b3b2b8b13537666d0775c9b1"
server_key_iv: "1ee129896df7a2227798fd7bd267be86"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.455677 18653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:12.461827 18653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:12.462013 18653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:12.462116 18653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:12.462270 18653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:12.462561 18653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:12.462595 18653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.462651 18653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:12.462679 18653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.467675 18653 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:40057
I20260620 19:21:12.467726 18780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:40057 every 8 connection(s)
I20260620 19:21:12.467953 18653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:21:12.472167 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.468449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:54113 (local address 127.1.1.254:34677)
0620 19:21:12.468499 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.468501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.468697 (+ 196us) server_negotiation.cc:407] Connection header received
0620 19:21:12.469106 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.469107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.469120 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.469161 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:12.469689 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.470113 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.470994 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.471182 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.471250 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.471924 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:12.471926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:12.471926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:12.471930 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:12.471942 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:12.471945 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:12.471985 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.472053 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.472075 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:21:12.472353 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.468454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34677 (local address 127.1.1.193:54113)
0620 19:21:12.468603 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.468638 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.469006 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.469213 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.469216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.469224 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:12.469611 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.469614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.470196 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.470197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.470909 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.470911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.471256 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.471268 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.471381 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.471796 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:12.471797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:12.471819 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:12.471836 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:12.471984 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:12.471989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:12.471991 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.472048 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.472149 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260620 19:21:12.472818 18781 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34677
I20260620 19:21:12.472886 18781 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:12.473101 18781 heartbeater.cc:511] Master 127.1.1.254:34677 requested a full tablet report, sending...
I20260620 19:21:12.473534 18593 ts_manager.cc:194] Registered new tserver with Master: 1065c295b6b04d60808739ece561803d (127.1.1.193:40057)
I20260620 19:21:12.473701 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18653
I20260620 19:21:12.473785 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:21:12.473944 1031 external_mini_cluster.cc:1428] Setting key 00e6d7699998929b1f1d4c472d5fe39b
I20260620 19:21:12.474231 18593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.193:54113
I20260620 19:21:12.474846 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--builtin_ntp_servers=127.1.1.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:12.544865 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:12.545019 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:12.545038 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:12.546010 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:12.546044 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:12.546065 18784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:12.547113 18784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:12.547246 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:12.547535 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:21:12.551651 18784 server_base.cc:1034] running on GCE node
W20260620 19:21:12.551712 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.551630 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.551820 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:12.552011 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:12.552209 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:12.553354 18784 hybrid_clock.cc:648] HybridClock initialized: now 1781983272553345 us; error 24 us; skew 500 ppm
I20260620 19:21:12.554379 18784 webserver.cc:460] Webserver started at http://127.1.1.194:34749/ using document root <none> and password file <none>
I20260620 19:21:12.554548 18784 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:12.554585 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:12.554661 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:12.555552 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "6aa224df97ac46a4b58ee5fb07a8cf68"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "31d182cd00cc9efadc87302093453ba8"
server_key_iv: "4324b8c79e4558f67e41439bad683faf"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.555859 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "6aa224df97ac46a4b58ee5fb07a8cf68"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "31d182cd00cc9efadc87302093453ba8"
server_key_iv: "4324b8c79e4558f67e41439bad683faf"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.556869 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:12.557657 18798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.557813 18784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:21:12.557896 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "6aa224df97ac46a4b58ee5fb07a8cf68"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "31d182cd00cc9efadc87302093453ba8"
server_key_iv: "4324b8c79e4558f67e41439bad683faf"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.557963 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:12.568598 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:12.568807 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:12.568926 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:12.569145 18784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:12.569404 18784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:12.569450 18784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.569484 18784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:12.569506 18784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.574501 18784 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:39397
I20260620 19:21:12.574535 18911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:39397 every 8 connection(s)
I20260620 19:21:12.574764 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:21:12.578617 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.575298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:42309 (local address 127.1.1.254:34677)
0620 19:21:12.575360 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.575362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.575507 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:21:12.575828 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.575829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.575838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.575877 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:12.576407 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.576800 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.577600 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.577752 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.577796 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.578431 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:12.578432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:12.578432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:12.578435 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:12.578443 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:12.578446 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:12.578479 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.578553 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.578568 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:21:12.578616 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18784
I20260620 19:21:12.578709 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:12.578846 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.575277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34677 (local address 127.1.1.194:42309)
0620 19:21:12.575418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.575442 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.575768 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.575939 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.575941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.575954 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:12.576353 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.576356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.576865 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.576866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.577539 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.577541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.577815 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.577815 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.577899 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.578330 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:12.578330 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:12.578353 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:12.578370 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:12.578485 (+ 115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:12.578488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:12.578491 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.578552 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.578644 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260620 19:21:12.578886 1031 external_mini_cluster.cc:1428] Setting key 1bfba8e72ae6b4d0f6ad1a0ab96f1182
I20260620 19:21:12.579342 18912 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34677
I20260620 19:21:12.579443 18912 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:12.579592 18912 heartbeater.cc:511] Master 127.1.1.254:34677 requested a full tablet report, sending...
I20260620 19:21:12.579767 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--builtin_ntp_servers=127.1.1.212:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:21:12.579926 18593 ts_manager.cc:194] Registered new tserver with Master: 6aa224df97ac46a4b58ee5fb07a8cf68 (127.1.1.194:39397)
I20260620 19:21:12.580299 18593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.194:42309
W20260620 19:21:12.653395 18915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:12.653625 18915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:12.653646 18915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:12.654644 18915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:12.654680 18915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:12.654726 18915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:12.655848 18915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:34677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:12.656045 18915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:12.656272 18915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:12.660772 18923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:12.660773 18920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:12.660869 18915 server_base.cc:1034] running on GCE node
W20260620 19:21:12.660826 18921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:12.661274 18915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:12.661442 18915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:12.662608 18915 hybrid_clock.cc:648] HybridClock initialized: now 1781983272662602 us; error 31 us; skew 500 ppm
I20260620 19:21:12.663734 18915 webserver.cc:460] Webserver started at http://127.1.1.195:42693/ using document root <none> and password file <none>
I20260620 19:21:12.663936 18915 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:12.663977 18915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:12.664111 18915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:12.665007 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "ad06ba7e78e14654891a5a5a377aa9ae"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "db1bab89da54ccc5b59f182df875b835"
server_key_iv: "823e7b0a04b137550611ea5583cc8823"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.665355 18915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "ad06ba7e78e14654891a5a5a377aa9ae"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "db1bab89da54ccc5b59f182df875b835"
server_key_iv: "823e7b0a04b137550611ea5583cc8823"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.666505 18915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:12.667095 18929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.667249 18915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:12.667325 18915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "ad06ba7e78e14654891a5a5a377aa9ae"
format_stamp: "Formatted at 2026-06-20 19:21:12 on dist-test-slave-mwkb"
server_key: "db1bab89da54ccc5b59f182df875b835"
server_key_iv: "823e7b0a04b137550611ea5583cc8823"
server_key_version: "encryptionkey@0"
I20260620 19:21:12.667369 18915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:12.683102 18915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:12.683295 18915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:12.683403 18915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:12.683540 18915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:12.683743 18915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:12.683768 18915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.683785 18915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:12.683838 18915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:12.688736 18915 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:35023
I20260620 19:21:12.688786 19042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:35023 every 8 connection(s)
I20260620 19:21:12.689059 18915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:21:12.693656 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.689870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:40565 (local address 127.1.1.254:34677)
0620 19:21:12.689928 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.689929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.690131 (+ 202us) server_negotiation.cc:407] Connection header received
0620 19:21:12.690497 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.690497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.690505 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.690552 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:12.691156 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.691614 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.692498 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.692671 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.692719 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.693429 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:12.693430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:12.693430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:12.693433 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:12.693445 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:12.693453 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:12.693509 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.693586 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.693604 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:21:12.693802 19045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.689838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:34677 (local address 127.1.1.195:40565)
0620 19:21:12.689994 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.690026 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.690423 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.690593 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.690595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.690602 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:12.691091 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.691094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.691689 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.691689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.692425 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.692427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.692749 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.692750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.692845 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.693294 (+ 449us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:12.693295 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:12.693329 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:12.693347 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:12.693521 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:12.693524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:12.693526 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.693570 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.693632 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 19:21:12.694092 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 18915
I20260620 19:21:12.694180 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:12.694279 19043 heartbeater.cc:347] Connected to a master server at 127.1.1.254:34677
I20260620 19:21:12.694360 19043 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:12.694378 1031 external_mini_cluster.cc:1428] Setting key f13181a3f07ee6ef9fb53207d25f921f
I20260620 19:21:12.694535 19043 heartbeater.cc:511] Master 127.1.1.254:34677 requested a full tablet report, sending...
I20260620 19:21:12.694888 18593 ts_manager.cc:194] Registered new tserver with Master: ad06ba7e78e14654891a5a5a377aa9ae (127.1.1.195:35023)
I20260620 19:21:12.695314 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:12.695319 18593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.195:40565
I20260620 19:21:12.698733 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.696639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48528 (local address 127.1.1.254:34677)
0620 19:21:12.696668 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.696670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.696767 (+ 97us) server_negotiation.cc:407] Connection header received
0620 19:21:12.696828 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.696828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.696834 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.696866 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:12.696973 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.697412 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.697991 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.698130 (+ 139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.698169 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.698592 (+ 423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:12.698593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:12.698593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:12.698596 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:12.698605 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:12.698607 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:12.698646 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.698683 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.698695 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:21:12.699941 18593 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:12.701439 18593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:12.706717 19051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35023 (local address 127.0.0.1:41656)
0620 19:21:12.703957 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.703964 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.704008 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.704299 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.704299 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.704301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.704335 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.704336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.705002 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.705003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.705753 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.705755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706455 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.706456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.706524 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.706536 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.706610 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.706659 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":79,"threads_started":1}
I20260620 19:21:12.706723 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.1.1.195:35023)
0620 19:21:12.703646 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.703650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.704187 (+ 537us) server_negotiation.cc:407] Connection header received
0620 19:21:12.704219 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.704221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.704232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.704303 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.704393 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.704918 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.705841 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706372 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.706447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.706514 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.706634 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.706662 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 19:21:12.707022 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40057 (local address 127.0.0.1:57958)
0620 19:21:12.703752 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.703784 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.703857 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.704057 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.704060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.704062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.704117 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.704119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.704928 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.704929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.706051 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.706054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706858 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.706859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.706907 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.706910 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.706933 (+ 23us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.706963 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":33,"thread_start_us":53,"threads_started":1}
I20260620 19:21:12.707504 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.1.1.193:40057)
0620 19:21:12.703754 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.703758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.703858 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:21:12.703937 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.703939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.703953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.704023 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.704173 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.704850 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.706144 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706773 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.707300 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.707380 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.707401 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.707434 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260620 19:21:12.707530 18977 tablet_service.cc:1467] Processing CreateTablet for tablet 181181180f444d858f562f2fedf89cf9 (DEFAULT_TABLE table=test-table [id=1e1eade14ab64ce08a181a791f35fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:12.707818 18977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181181180f444d858f562f2fedf89cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:12.707914 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:39397 (local address 127.0.0.1:53852)
0620 19:21:12.704071 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.704078 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.704114 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.705060 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.705061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.705063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.705174 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.705175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706202 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.706203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.707087 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.707089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.707773 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.707774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.707812 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.707815 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.707853 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.707876 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":56,"thread_start_us":64,"threads_started":1}
I20260620 19:21:12.708000 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.703563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.1.1.194:39397)
0620 19:21:12.703711 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.703714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.704116 (+ 402us) server_negotiation.cc:407] Connection header received
0620 19:21:12.704972 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.704974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.704989 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.705051 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.705239 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.706135 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.707157 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.707625 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.707682 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.707747 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.707892 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.707935 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260620 19:21:12.708554 18846 tablet_service.cc:1467] Processing CreateTablet for tablet 181181180f444d858f562f2fedf89cf9 (DEFAULT_TABLE table=test-table [id=1e1eade14ab64ce08a181a791f35fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:12.708757 18846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181181180f444d858f562f2fedf89cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:12.709424 18715 tablet_service.cc:1467] Processing CreateTablet for tablet 181181180f444d858f562f2fedf89cf9 (DEFAULT_TABLE table=test-table [id=1e1eade14ab64ce08a181a791f35fdd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:12.709697 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181181180f444d858f562f2fedf89cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:12.709997 19057 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Bootstrap starting.
I20260620 19:21:12.710467 19057 tablet_bootstrap.cc:654] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:12.710651 19057 log.cc:824] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:12.711210 19057 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: No bootstrap required, opened a new log
I20260620 19:21:12.711272 19057 ts_tablet_manager.cc:1397] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:12.711409 19059 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Bootstrap starting.
I20260620 19:21:12.711997 19059 tablet_bootstrap.cc:654] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:12.712309 19060 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Bootstrap starting.
I20260620 19:21:12.712344 19059 log.cc:824] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:12.712896 19060 tablet_bootstrap.cc:654] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:12.713239 19060 log.cc:824] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:12.713585 19059 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: No bootstrap required, opened a new log
I20260620 19:21:12.713653 19059 ts_tablet_manager.cc:1397] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:12.713891 19060 tablet_bootstrap.cc:492] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: No bootstrap required, opened a new log
I20260620 19:21:12.713928 19060 ts_tablet_manager.cc:1397] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:21:12.713904 19057 raft_consensus.cc:348] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.714057 19057 raft_consensus.cc:374] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:12.714107 19057 raft_consensus.cc:729] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad06ba7e78e14654891a5a5a377aa9ae, State: Initialized, Role: FOLLOWER
I20260620 19:21:12.714216 19057 consensus_queue.cc:260] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.714509 19057 ts_tablet_manager.cc:1428] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:21:12.714569 19043 heartbeater.cc:503] Master 127.1.1.254:34677 was elected leader, sending a full tablet report...
I20260620 19:21:12.715826 19059 raft_consensus.cc:348] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.715914 19059 raft_consensus.cc:374] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:12.715934 19059 raft_consensus.cc:729] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa224df97ac46a4b58ee5fb07a8cf68, State: Initialized, Role: FOLLOWER
I20260620 19:21:12.715909 19060 raft_consensus.cc:348] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.716010 19060 raft_consensus.cc:374] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:12.716028 19060 raft_consensus.cc:729] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1065c295b6b04d60808739ece561803d, State: Initialized, Role: FOLLOWER
I20260620 19:21:12.716024 19059 consensus_queue.cc:260] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.716092 19060 consensus_queue.cc:260] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.716230 19059 ts_tablet_manager.cc:1428] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:21:12.716252 18912 heartbeater.cc:503] Master 127.1.1.254:34677 was elected leader, sending a full tablet report...
I20260620 19:21:12.716282 19060 ts_tablet_manager.cc:1428] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:21:12.716300 18781 heartbeater.cc:503] Master 127.1.1.254:34677 was elected leader, sending a full tablet report...
W20260620 19:21:12.718343 18782 tablet.cc:2369] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:12.725247 19064 raft_consensus.cc:482] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:12.725314 19064 raft_consensus.cc:504] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.725617 19064 leader_election.cc:288] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1065c295b6b04d60808739ece561803d (127.1.1.193:40057), ad06ba7e78e14654891a5a5a377aa9ae (127.1.1.195:35023)
I20260620 19:21:12.727957 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.725773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:38233 (local address 127.1.1.195:35023)
0620 19:21:12.725811 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.725812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.725909 (+ 97us) server_negotiation.cc:407] Connection header received
0620 19:21:12.725986 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.725988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.725995 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.726043 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.726207 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.726610 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727396 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.727664 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727735 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.727741 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.727876 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.727903 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:21:12.727963 19056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.725820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:51371 (local address 127.1.1.193:40057)
0620 19:21:12.725847 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.725848 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.725855 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:12.725872 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.725873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.725880 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.725929 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:12.726081 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.726561 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727366 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.727662 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727735 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.727741 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.727879 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.727907 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:21:12.727977 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.725718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40057 (local address 127.1.1.194:51371)
0620 19:21:12.725768 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.725777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.725812 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.725965 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.725966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.725967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.726018 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.726019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.726629 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.726630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.727288 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.727290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.727744 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.727805 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.727807 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.727861 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.727903 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260620 19:21:12.727989 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.725718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:35023 (local address 127.1.1.194:38233)
0620 19:21:12.725864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.725872 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:12.725926 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:12.726073 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:12.726074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:12.726076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:12.726146 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.726147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.726689 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.726690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.727298 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:12.727299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.727737 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.727737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:12.727795 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.727798 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:12.727858 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:21:12.727898 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":12,"thread_start_us":61,"threads_started":1}
I20260620 19:21:12.728238 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181181180f444d858f562f2fedf89cf9" candidate_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1065c295b6b04d60808739ece561803d" is_pre_election: true
I20260620 19:21:12.728267 18997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181181180f444d858f562f2fedf89cf9" candidate_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" is_pre_election: true
I20260620 19:21:12.728374 18997 raft_consensus.cc:2454] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aa224df97ac46a4b58ee5fb07a8cf68 in term 0.
I20260620 19:21:12.728374 18735 raft_consensus.cc:2454] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aa224df97ac46a4b58ee5fb07a8cf68 in term 0.
I20260620 19:21:12.728557 18802 leader_election.cc:302] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1065c295b6b04d60808739ece561803d, 6aa224df97ac46a4b58ee5fb07a8cf68; no voters:
I20260620 19:21:12.728649 19064 raft_consensus.cc:2781] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:12.728694 19064 raft_consensus.cc:482] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:12.728715 19064 raft_consensus.cc:3037] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.729225 19064 raft_consensus.cc:504] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.729336 19064 leader_election.cc:288] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [CANDIDATE]: Term 1 election: Requested vote from peers 1065c295b6b04d60808739ece561803d (127.1.1.193:40057), ad06ba7e78e14654891a5a5a377aa9ae (127.1.1.195:35023)
I20260620 19:21:12.729489 18997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181181180f444d858f562f2fedf89cf9" candidate_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad06ba7e78e14654891a5a5a377aa9ae"
I20260620 19:21:12.729485 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181181180f444d858f562f2fedf89cf9" candidate_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1065c295b6b04d60808739ece561803d"
I20260620 19:21:12.729552 18997 raft_consensus.cc:3037] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.729552 18735 raft_consensus.cc:3037] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:12.730252 18735 raft_consensus.cc:2454] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aa224df97ac46a4b58ee5fb07a8cf68 in term 1.
I20260620 19:21:12.730258 18997 raft_consensus.cc:2454] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aa224df97ac46a4b58ee5fb07a8cf68 in term 1.
I20260620 19:21:12.730412 18801 leader_election.cc:302] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aa224df97ac46a4b58ee5fb07a8cf68, ad06ba7e78e14654891a5a5a377aa9ae; no voters:
I20260620 19:21:12.730509 19064 raft_consensus.cc:2781] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:12.730633 19064 raft_consensus.cc:686] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 1 LEADER]: Becoming Leader. State: Replica: 6aa224df97ac46a4b58ee5fb07a8cf68, State: Running, Role: LEADER
I20260620 19:21:12.730715 19064 consensus_queue.cc:237] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } }
I20260620 19:21:12.731310 18593 catalog_manager.cc:5495] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aa224df97ac46a4b58ee5fb07a8cf68 (127.1.1.194). New cstate: current_term: 1 leader_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aa224df97ac46a4b58ee5fb07a8cf68" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 39397 } health_report { overall_health: HEALTHY } } }
I20260620 19:21:12.735692 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.733743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.1.1.194:39397)
0620 19:21:12.733777 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:12.733778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:12.733787 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:12.733815 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:12.733815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:12.733827 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:12.733874 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:12.733952 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.734312 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.734935 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:12.735153 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:12.735223 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:12.735380 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:12.735467 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:12.735545 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:12.735595 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:21:12.735609 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:21:12.737015 18735 raft_consensus.cc:1261] T 181181180f444d858f562f2fedf89cf9 P 1065c295b6b04d60808739ece561803d [term 1 FOLLOWER]: Refusing update from remote peer 6aa224df97ac46a4b58ee5fb07a8cf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:12.737016 18997 raft_consensus.cc:1261] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 1 FOLLOWER]: Refusing update from remote peer 6aa224df97ac46a4b58ee5fb07a8cf68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:12.737300 19067 consensus_queue.cc:1035] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1065c295b6b04d60808739ece561803d" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 40057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:12.737416 19064 consensus_queue.cc:1035] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad06ba7e78e14654891a5a5a377aa9ae" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:12.739777 19071 mvcc.cc:205] Tried to move back new op lower bound from 7299003485128777728 to 7299003485105512448. Current Snapshot: MvccSnapshot[applied={T|T < 7299003485128777728}]
I20260620 19:21:12.739912 19072 mvcc.cc:205] Tried to move back new op lower bound from 7299003485128777728 to 7299003485105512448. Current Snapshot: MvccSnapshot[applied={T|T < 7299003485128777728}]
I20260620 19:21:12.741485 18592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:12.741559 18592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:12.742174 18592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e2705b1aa7843929a1a1cd9c2448307: Sending DeleteTablet for 3 replicas of tablet 181181180f444d858f562f2fedf89cf9
I20260620 19:21:12.742486 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet 181181180f444d858f562f2fedf89cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='slave'} at 127.0.0.1:57958
I20260620 19:21:12.742548 18846 tablet_service.cc:1514] Processing DeleteTablet for tablet 181181180f444d858f562f2fedf89cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='slave'} at 127.0.0.1:53852
I20260620 19:21:12.742689 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18653
I20260620 19:21:12.742694 18977 tablet_service.cc:1514] Processing DeleteTablet for tablet 181181180f444d858f562f2fedf89cf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:12 UTC) from {username='slave'} at 127.0.0.1:41656
I20260620 19:21:12.744652 19081 tablet_replica.cc:330] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: stopping tablet replica
I20260620 19:21:12.744762 19081 raft_consensus.cc:2229] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:12.744844 19081 raft_consensus.cc:2258] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:12.745226 19081 ts_tablet_manager.cc:1905] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:12.746883 19081 ts_tablet_manager.cc:1918] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:12.746973 19081 log.cc:1196] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/181181180f444d858f562f2fedf89cf9
I20260620 19:21:12.747100 19080 tablet_replica.cc:330] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: stopping tablet replica
I20260620 19:21:12.747223 19080 raft_consensus.cc:2229] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:21:12.747222 19081 ts_tablet_manager.cc:1939] T 181181180f444d858f562f2fedf89cf9 P ad06ba7e78e14654891a5a5a377aa9ae: Deleting consensus metadata
I20260620 19:21:12.747321 19080 raft_consensus.cc:2258] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:12.747629 18579 catalog_manager.cc:4841] TS ad06ba7e78e14654891a5a5a377aa9ae (127.1.1.195:35023): tablet 181181180f444d858f562f2fedf89cf9 (table test-table [id=1e1eade14ab64ce08a181a791f35fdd0]) successfully deleted
I20260620 19:21:12.747766 19080 ts_tablet_manager.cc:1905] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:21:12.748926 18580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:40057 (error 108)
I20260620 19:21:12.749119 19080 ts_tablet_manager.cc:1918] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:12.749200 19080 log.cc:1196] T 181181180f444d858f562f2fedf89cf9 P 6aa224df97ac46a4b58ee5fb07a8cf68: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/181181180f444d858f562f2fedf89cf9
I20260620 19:21:12.749197 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18784
I20260620 19:21:12.749434 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.749345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:40057 (local address 127.0.0.1:57966)
0620 19:21:12.749379 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.749406 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:40057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 19:21:12.749517 18580 catalog_manager.cc:4584] TS 1065c295b6b04d60808739ece561803d (127.1.1.193:40057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 181181180f444d858f562f2fedf89cf9: Network error: Client connection negotiation failed: client connection to 127.1.1.193:40057: connect: Connection refused (error 111)
I20260620 19:21:12.754287 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:12.754139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:39397 (local address 127.0.0.1:53872)
0620 19:21:12.754220 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:12.754254 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.194:39397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 19:21:12.754370 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18915
I20260620 19:21:12.758405 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 18562
2026-06-20T19:21:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (547 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:12 dist-test-slave-mwkb krb5kdc[19085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:12 dist-test-slave-mwkb krb5kdc[19085](info): set up 2 sockets
Jun 20 19:21:12 dist-test-slave-mwkb krb5kdc[19085](info): commencing operation
krb5kdc: starting...
W20260620 19:21:13.692595 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:13 dist-test-slave-mwkb krb5kdc[19085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:13.788424 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46599
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:13.859661 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:13.859882 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:13.859920 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:13.860980 19098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:13.861045 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:13.861068 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:13.861104 19098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:13.861131 19098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:13.862390 19098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:13.862560 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:13.862705 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:13.866773 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:13.866827 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:13.866986 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:13.867002 19098 server_base.cc:1034] running on GCE node
I20260620 19:21:13.867202 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:13.867452 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:13.868625 19098 hybrid_clock.cc:648] HybridClock initialized: now 1781983273868614 us; error 28 us; skew 500 ppm
Jun 20 19:21:13 dist-test-slave-mwkb krb5kdc[19085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983273, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:13.870654 19098 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:13.871039 19098 webserver.cc:460] Webserver started at http://127.1.1.254:40017/ using document root <none> and password file <none>
I20260620 19:21:13.871248 19098 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:13.871301 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:13.871479 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:13.872316 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "429e76bd780d47e7888c394eee5d0b06"
format_stamp: "Formatted at 2026-06-20 19:21:13 on dist-test-slave-mwkb"
server_key: "9830a3705062fd1a46a0e141b1bd03bb"
server_key_iv: "c297d7f51b43790ac3f3cd0f36b4a65f"
server_key_version: "encryptionkey@0"
I20260620 19:21:13.872661 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "429e76bd780d47e7888c394eee5d0b06"
format_stamp: "Formatted at 2026-06-20 19:21:13 on dist-test-slave-mwkb"
server_key: "9830a3705062fd1a46a0e141b1bd03bb"
server_key_iv: "c297d7f51b43790ac3f3cd0f36b4a65f"
server_key_version: "encryptionkey@0"
I20260620 19:21:13.873975 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:13.874585 19113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:13.874732 19098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:13.874814 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "429e76bd780d47e7888c394eee5d0b06"
format_stamp: "Formatted at 2026-06-20 19:21:13 on dist-test-slave-mwkb"
server_key: "9830a3705062fd1a46a0e141b1bd03bb"
server_key_iv: "c297d7f51b43790ac3f3cd0f36b4a65f"
server_key_version: "encryptionkey@0"
I20260620 19:21:13.874884 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:13.890918 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:13.892969 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:13.893169 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:13.896756 19098 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46599
I20260620 19:21:13.896791 19165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46599 every 8 connection(s)
I20260620 19:21:13.897130 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:13.897814 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:13.899585 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: Bootstrap starting.
I20260620 19:21:13.900050 19166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:13.900267 19166 log.cc:824] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:13.900830 19166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: No bootstrap required, opened a new log
I20260620 19:21:13.901674 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19098
I20260620 19:21:13.901768 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:13.901933 1031 external_mini_cluster.cc:1428] Setting key b21a895a7a48d7306c8acb6b9b972991
I20260620 19:21:13.902732 19166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } }
I20260620 19:21:13.902906 19166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:13.902945 19166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 429e76bd780d47e7888c394eee5d0b06, State: Initialized, Role: FOLLOWER
I20260620 19:21:13.903075 19166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } }
I20260620 19:21:13.903177 19166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:13.903216 19166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:13.903267 19166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:13.904230 19166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } }
I20260620 19:21:13.904327 19166 leader_election.cc:302] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 429e76bd780d47e7888c394eee5d0b06; no voters:
I20260620 19:21:13.904461 19166 leader_election.cc:288] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:13.904520 19171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:13.904644 19166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:13.904644 19171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [term 1 LEADER]: Becoming Leader. State: Replica: 429e76bd780d47e7888c394eee5d0b06, State: Running, Role: LEADER
I20260620 19:21:13.904843 19171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } }
I20260620 19:21:13.906080 19173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "429e76bd780d47e7888c394eee5d0b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } } }
I20260620 19:21:13.906092 19172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 429e76bd780d47e7888c394eee5d0b06. Latest consensus state: current_term: 1 leader_uuid: "429e76bd780d47e7888c394eee5d0b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "429e76bd780d47e7888c394eee5d0b06" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46599 } } }
I20260620 19:21:13.906267 19172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:13.906252 19173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06 [sys.catalog]: This master's current role is: LEADER
W20260620 19:21:13.906522 19186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:13.906582 19186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 19:21:13 dist-test-slave-mwkb krb5kdc[19085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:13.909191 19188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:13.909374 19188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:13.911078 19188 catalog_manager.cc:1347] Generated new cluster ID: 6c848578f6924394a73d57fec5cb2f80
I20260620 19:21:13.911154 19188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:13.912500 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:13.902370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.1.1.254:46599)
0620 19:21:13.902548 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:13.902558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:13.902586 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:21:13.902691 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:13.902697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:13.902748 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:13.902885 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:13.903627 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:13.904857 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:13.905858 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:13.906446 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:13.906539 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:13.908780 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:13.908784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:13.908785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:13.908794 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:13.910649 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:13.910937 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:13.910940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:13.910940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:13.911058 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:13.911269 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:13.911270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:13.911270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:13.911563 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:13.911690 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:13.912114 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 19:21:13.912307 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260620 19:21:13.915051 19188 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:13.915637 19188 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:13.917832 19188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: Generated new TSK 0
I20260620 19:21:13.917973 19188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:21:13.939689 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:14.012586 19192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:14.012764 19192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:14.012825 19192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:14.014056 19192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:14.014098 19192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:14.014124 19192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:14.015503 19192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:14.015704 19192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:14.015919 19192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:14.021106 19197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.021121 19198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.021166 19200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.021323 19192 server_base.cc:1034] running on GCE node
I20260620 19:21:14.021592 19192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:14.021840 19192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:14.022993 19192 hybrid_clock.cc:648] HybridClock initialized: now 1781983274022981 us; error 40 us; skew 500 ppm
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:14.025200 19192 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:21:14.025588 19192 webserver.cc:460] Webserver started at http://127.1.1.193:33363/ using document root <none> and password file <none>
I20260620 19:21:14.025790 19192 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:14.025851 19192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:14.025970 19192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:14.026895 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "32812968a6d84abbbd50fa22f1ff4043"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "8298700653433da6e46e1e7abef06bb0"
server_key_iv: "66eaade9db7ff0861894ad2b7b05bf88"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.027274 19192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "32812968a6d84abbbd50fa22f1ff4043"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "8298700653433da6e46e1e7abef06bb0"
server_key_iv: "66eaade9db7ff0861894ad2b7b05bf88"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.028422 19192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:14.029021 19207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.029177 19192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.029304 19192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "32812968a6d84abbbd50fa22f1ff4043"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "8298700653433da6e46e1e7abef06bb0"
server_key_iv: "66eaade9db7ff0861894ad2b7b05bf88"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.029374 19192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:14.040963 19192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:14.042495 19192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:14.042644 19192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:14.042852 19192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:14.043126 19192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:14.043179 19192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.043218 19192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:14.043256 19192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.048282 19192 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:34925
I20260620 19:21:14.048342 19320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:34925 every 8 connection(s)
I20260620 19:21:14.048604 19192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:21:14.053362 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19192
I20260620 19:21:14.053457 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:14.053658 1031 external_mini_cluster.cc:1428] Setting key a8b25a2c7969178cce44345094da419a
I20260620 19:21:14.056680 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.049294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:51043 (local address 127.1.1.254:46599)
0620 19:21:14.049335 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.049336 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.049497 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:21:14.049861 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.049861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.049869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.049915 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.050646 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.051076 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.051904 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.052072 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.052119 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.054031 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.054032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.054032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:14.054038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.055361 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.055832 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.055833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.055833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.055916 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.056128 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.056129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.056129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.056265 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.056331 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.056544 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.056622 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:14.056768 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.049257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46599 (local address 127.1.1.193:51043)
0620 19:21:14.049390 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.049424 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.049792 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.049956 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.049958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.050421 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:14.050576 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.050579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.051155 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.051156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.051834 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.051836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.052137 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.052138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.052225 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.052619 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:14.052627 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:14.053962 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:14.055466 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.055474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.055477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.055745 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.055982 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.055983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.055983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.056069 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.056331 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:14.056335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:14.056427 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.056500 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.056597 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260620 19:21:14.057268 19321 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46599
I20260620 19:21:14.057354 19321 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:14.057509 19321 heartbeater.cc:511] Master 127.1.1.254:46599 requested a full tablet report, sending...
I20260620 19:21:14.057889 19130 ts_manager.cc:194] Registered new tserver with Master: 32812968a6d84abbbd50fa22f1ff4043 (127.1.1.193:34925)
I20260620 19:21:14.058399 19130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:51043
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:21:14.075397 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:14.146929 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:14.147096 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:14.147117 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:14.148066 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:14.148105 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:14.148128 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:14.149437 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:14.149578 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:14.149715 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:14.153887 19332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.153904 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.153908 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.154223 19326 server_base.cc:1034] running on GCE node
I20260620 19:21:14.154404 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:14.154633 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:14.155784 19326 hybrid_clock.cc:648] HybridClock initialized: now 1781983274155759 us; error 37 us; skew 500 ppm
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:14.157832 19326 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:21:14.158206 19326 webserver.cc:460] Webserver started at http://127.1.1.194:32837/ using document root <none> and password file <none>
I20260620 19:21:14.158407 19326 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:14.158475 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:14.158615 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:14.159508 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "d740b4dbb36c4030ae8b9444f324e2f6"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "b01eb11a5824ed13f86ea8128cc18e17"
server_key_iv: "596f168a0eb6995203346e906a560f10"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.159832 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "d740b4dbb36c4030ae8b9444f324e2f6"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "b01eb11a5824ed13f86ea8128cc18e17"
server_key_iv: "596f168a0eb6995203346e906a560f10"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.160989 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:14.161599 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.161726 19326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.161808 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "d740b4dbb36c4030ae8b9444f324e2f6"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "b01eb11a5824ed13f86ea8128cc18e17"
server_key_iv: "596f168a0eb6995203346e906a560f10"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.161895 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:14.175127 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:14.176566 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:14.176697 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:14.176870 19326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:14.177172 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:14.177243 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.177281 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:14.177296 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.182518 19326 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:37507
I20260620 19:21:14.182580 19454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:37507 every 8 connection(s)
I20260620 19:21:14.182853 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:14.189265 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19326
I20260620 19:21:14.189365 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:14.189536 1031 external_mini_cluster.cc:1428] Setting key 9a349b30720ec739d2448238a6eba43d
I20260620 19:21:14.190712 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.183469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46599 (local address 127.1.1.194:46055)
0620 19:21:14.183630 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.183660 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.184033 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.184209 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.184212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.184657 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:14.184809 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.184812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.185453 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.185454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.186106 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.186108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.186389 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.186390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.186471 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.186794 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:14.186800 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:14.187848 (+ 1048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:14.189413 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.189418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.189420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.189681 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.189921 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.189922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.189922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.190043 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.190304 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:14.190308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:14.190400 (+ 92us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.190475 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.190570 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260620 19:21:14.190742 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.183515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:46055 (local address 127.1.1.254:46599)
0620 19:21:14.183597 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.183598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.183727 (+ 129us) server_negotiation.cc:407] Connection header received
0620 19:21:14.184103 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.184103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.184112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.184158 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.184875 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.185384 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.186169 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.186330 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.186371 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.187920 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.187921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.187921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:14.187926 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.189342 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.189767 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.189767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.189768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.189854 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.190105 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.190106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.190106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.190237 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.190309 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.190514 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.190591 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:21:14.191105 19455 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46599
I20260620 19:21:14.191195 19455 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:14.191370 19455 heartbeater.cc:511] Master 127.1.1.254:46599 requested a full tablet report, sending...
I20260620 19:21:14.191774 19130 ts_manager.cc:194] Registered new tserver with Master: d740b4dbb36c4030ae8b9444f324e2f6 (127.1.1.194:37507)
I20260620 19:21:14.192291 19130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:46055
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:21:14.211680 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:14.281742 19460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:14.281963 19460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:14.282011 19460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:14.283107 19460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:14.283170 19460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:14.283214 19460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:14.284444 19460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:46599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:14.284662 19460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:14.284865 19460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:14.288923 19468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.288936 19466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.289026 19465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.289407 19460 server_base.cc:1034] running on GCE node
I20260620 19:21:14.289585 19460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:14.289821 19460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:14.290952 19460 hybrid_clock.cc:648] HybridClock initialized: now 1781983274290925 us; error 41 us; skew 500 ppm
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:14.293020 19460 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:21:14.293411 19460 webserver.cc:460] Webserver started at http://127.1.1.195:45209/ using document root <none> and password file <none>
I20260620 19:21:14.293584 19460 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:14.293619 19460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:14.293691 19460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:14.294484 19460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "5eedcecaefd849abae52c297b7e8a9b9"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "e888ef004d0a00558d8e2dd137cbcfcd"
server_key_iv: "6c02481ce2418e3e21c350b0e7e5693c"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.294735 19460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "5eedcecaefd849abae52c297b7e8a9b9"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "e888ef004d0a00558d8e2dd137cbcfcd"
server_key_iv: "6c02481ce2418e3e21c350b0e7e5693c"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.295786 19460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:14.296329 19475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.296458 19460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:14.296531 19460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "5eedcecaefd849abae52c297b7e8a9b9"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "e888ef004d0a00558d8e2dd137cbcfcd"
server_key_iv: "6c02481ce2418e3e21c350b0e7e5693c"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.296576 19460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:14.321462 19460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:14.322938 19460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:14.323038 19460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:14.323185 19460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:14.323390 19460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:14.323417 19460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.323436 19460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:14.323448 19460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.328351 19460 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:44459
I20260620 19:21:14.328413 19588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:44459 every 8 connection(s)
I20260620 19:21:14.328698 19460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:21:14 dist-test-slave-mwkb krb5kdc[19085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983274, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:14.335532 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19460
I20260620 19:21:14.335638 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:14.335804 1031 external_mini_cluster.cc:1428] Setting key c2a2c52a67202a7fa7a407fb1de1e5e7
I20260620 19:21:14.336300 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.329380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:48601 (local address 127.1.1.254:46599)
0620 19:21:14.329426 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.329429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.329622 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:21:14.329974 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.329974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.329983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.330026 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.330733 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.331170 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.332101 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.332272 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.332340 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.334170 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.334171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.334171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:14.334177 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.335140 (+ 963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.335468 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.335469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.335470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.335526 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.335737 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.335737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.335738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.335833 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.335896 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.336131 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.336203 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:14.336390 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.329364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:46599 (local address 127.1.1.195:48601)
0620 19:21:14.329517 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.329547 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.329900 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.330066 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.330068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.330523 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:14.330669 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.330673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.331239 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.331239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.332026 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.332028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.332357 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.332357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.332492 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.332869 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:14.332876 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:14.334100 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:14.335199 (+ 1099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.335204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.335205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.335378 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.335585 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:14.335585 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:14.335586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:14.335671 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:14.335912 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:14.335918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:14.336014 (+ 96us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.336100 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.336220 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260620 19:21:14.336786 19589 heartbeater.cc:347] Connected to a master server at 127.1.1.254:46599
I20260620 19:21:14.336849 19589 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:14.337003 19589 heartbeater.cc:511] Master 127.1.1.254:46599 requested a full tablet report, sending...
I20260620 19:21:14.337332 19130 ts_manager.cc:194] Registered new tserver with Master: 5eedcecaefd849abae52c297b7e8a9b9 (127.1.1.195:44459)
I20260620 19:21:14.337603 19130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:48601
I20260620 19:21:14.347131 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:14.353641 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.348315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.1.1.254:46599)
0620 19:21:14.348369 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.348370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.348466 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:21:14.348505 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.348505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.348525 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.348568 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.349054 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.349557 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.350219 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.350380 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.350424 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.351457 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.351458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.351459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:14.351482 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.352715 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.353036 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.353037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.353037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.353128 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:14.353273 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:14.353273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:14.353273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:14.353378 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.353451 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.353544 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.353587 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:21:14.355008 19130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:14.356294 19130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:14.361829 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.1.1.195:44459)
0620 19:21:14.358895 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.358898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.358905 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:14.358930 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.358931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.358942 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.359014 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:14.359165 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.359926 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361061 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.361461 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361546 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.361591 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.361725 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.361762 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":84,"threads_started":1}
I20260620 19:21:14.361848 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:44459 (local address 127.0.0.1:48740)
0620 19:21:14.358383 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.358453 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.358493 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.359072 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.359072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.359074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:14.359107 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.359108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.360023 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.360024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.361003 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.361005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.361551 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.361623 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.361633 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.361712 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.361769 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":20,"thread_start_us":49,"threads_started":1}
I20260620 19:21:14.361850 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37507 (local address 127.0.0.1:48688)
0620 19:21:14.358382 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.358418 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.358480 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.358679 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.358681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.358682 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:14.358719 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.358720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.359879 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.359880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.360772 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.360774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.361607 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.361659 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.361663 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.361702 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.361766 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":3,"spinlock_wait_cycles":384,"thread_start_us":89,"threads_started":1}
I20260620 19:21:14.361984 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.1.1.194:37507)
0620 19:21:14.358490 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.358494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.358530 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:21:14.358586 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.358587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.358598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.358663 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:14.358795 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.359781 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.360855 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.361476 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361533 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.361578 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.361884 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.361920 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
I20260620 19:21:14.362536 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34925 (local address 127.0.0.1:56956)
0620 19:21:14.358735 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.358742 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.358805 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.359407 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.359408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.359410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:14.359443 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.359444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.360634 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.360635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.361553 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.361556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.362280 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.362281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.362367 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.362371 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.362423 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.362457 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":63,"threads_started":1}
I20260620 19:21:14.362818 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.358361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.1.1.193:34925)
0620 19:21:14.359228 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.359232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.359239 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:14.359282 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.359300 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.359314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.359387 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:14.359503 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.360538 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.361629 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.362197 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.362250 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.362278 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.362696 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.362741 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":78,"threads_started":1}
I20260620 19:21:14.362936 19523 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 (DEFAULT_TABLE table=test-table [id=17cd52cf31dc4a6eb22024dd7260adfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:14.362943 19389 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 (DEFAULT_TABLE table=test-table [id=17cd52cf31dc4a6eb22024dd7260adfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:14.363246 19389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb6bd5dca06469b83a6a612ae076c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:14.363252 19523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb6bd5dca06469b83a6a612ae076c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:14.365525 19604 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Bootstrap starting.
I20260620 19:21:14.365520 19255 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 (DEFAULT_TABLE table=test-table [id=17cd52cf31dc4a6eb22024dd7260adfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:14.365800 19255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb6bd5dca06469b83a6a612ae076c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:14.366031 19604 tablet_bootstrap.cc:654] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:14.366379 19604 log.cc:824] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:14.366592 19603 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Bootstrap starting.
I20260620 19:21:14.366972 19603 tablet_bootstrap.cc:654] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:14.367000 19604 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: No bootstrap required, opened a new log
I20260620 19:21:14.367051 19604 ts_tablet_manager.cc:1397] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:14.367300 19603 log.cc:824] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:14.367818 19603 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: No bootstrap required, opened a new log
I20260620 19:21:14.367854 19603 ts_tablet_manager.cc:1397] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:14.368265 19607 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Bootstrap starting.
I20260620 19:21:14.368786 19607 tablet_bootstrap.cc:654] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:14.368935 19604 raft_consensus.cc:348] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.369073 19604 raft_consensus.cc:374] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:14.369151 19604 raft_consensus.cc:729] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d740b4dbb36c4030ae8b9444f324e2f6, State: Initialized, Role: FOLLOWER
I20260620 19:21:14.369253 19604 consensus_queue.cc:260] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.369304 19607 log.cc:824] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:14.369521 19455 heartbeater.cc:503] Master 127.1.1.254:46599 was elected leader, sending a full tablet report...
I20260620 19:21:14.369518 19604 ts_tablet_manager.cc:1428] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:21:14.369544 19603 raft_consensus.cc:348] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.369684 19603 raft_consensus.cc:374] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:14.369714 19603 raft_consensus.cc:729] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eedcecaefd849abae52c297b7e8a9b9, State: Initialized, Role: FOLLOWER
I20260620 19:21:14.369841 19603 consensus_queue.cc:260] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.369972 19607 tablet_bootstrap.cc:492] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: No bootstrap required, opened a new log
I20260620 19:21:14.370031 19607 ts_tablet_manager.cc:1397] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:14.370071 19603 ts_tablet_manager.cc:1428] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:14.370080 19589 heartbeater.cc:503] Master 127.1.1.254:46599 was elected leader, sending a full tablet report...
I20260620 19:21:14.371440 19607 raft_consensus.cc:348] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.371536 19607 raft_consensus.cc:374] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:14.371553 19607 raft_consensus.cc:729] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32812968a6d84abbbd50fa22f1ff4043, State: Initialized, Role: FOLLOWER
I20260620 19:21:14.371660 19607 consensus_queue.cc:260] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.371883 19607 ts_tablet_manager.cc:1428] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:14.371883 19321 heartbeater.cc:503] Master 127.1.1.254:46599 was elected leader, sending a full tablet report...
W20260620 19:21:14.433331 19456 tablet.cc:2369] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:21:14.549245 19322 tablet.cc:2369] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:21:14.579175 19590 tablet.cc:2369] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:14.588783 19610 raft_consensus.cc:482] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:14.588861 19610 raft_consensus.cc:504] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.589233 19610 leader_election.cc:288] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d740b4dbb36c4030ae8b9444f324e2f6 (127.1.1.194:37507), 32812968a6d84abbbd50fa22f1ff4043 (127.1.1.193:34925)
I20260620 19:21:14.591637 19591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.589367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37507 (local address 127.1.1.195:41637)
0620 19:21:14.589408 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.589419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.589457 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.589606 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.589607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.589609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:14.589677 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.589678 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.590228 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.590229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.590847 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.590849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.591346 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.591347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.591461 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.591464 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.591530 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.591572 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":3}
I20260620 19:21:14.591635 19601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.589412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:41637 (local address 127.1.1.194:37507)
0620 19:21:14.589480 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.589482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.589491 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:14.589516 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.589517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.589528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.589598 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:14.589732 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.590154 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.590931 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.591249 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.591346 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.591355 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.591536 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.591573 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:21:14.591769 19602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.589412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:49571 (local address 127.1.1.193:34925)
0620 19:21:14.589454 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.589455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.589639 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:21:14.589700 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.589701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.589708 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.589764 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:14.589902 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.590286 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.591068 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.591429 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.591522 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.591531 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.591678 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.591712 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:14.591782 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.589372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34925 (local address 127.1.1.195:49571)
0620 19:21:14.589596 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.589603 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.589642 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.589772 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.589773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.589775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:14.589840 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.589841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.590361 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.590362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.590987 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.590989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.591529 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.591530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.591599 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.591603 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.591666 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.591705 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":62,"thread_start_us":91,"threads_started":1}
I20260620 19:21:14.591991 19409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb6bd5dca06469b83a6a612ae076c56" candidate_uuid: "5eedcecaefd849abae52c297b7e8a9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" is_pre_election: true
I20260620 19:21:14.592046 19275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb6bd5dca06469b83a6a612ae076c56" candidate_uuid: "5eedcecaefd849abae52c297b7e8a9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32812968a6d84abbbd50fa22f1ff4043" is_pre_election: true
I20260620 19:21:14.592120 19409 raft_consensus.cc:2454] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eedcecaefd849abae52c297b7e8a9b9 in term 0.
I20260620 19:21:14.592208 19275 raft_consensus.cc:2454] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eedcecaefd849abae52c297b7e8a9b9 in term 0.
I20260620 19:21:14.592351 19476 leader_election.cc:302] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eedcecaefd849abae52c297b7e8a9b9, d740b4dbb36c4030ae8b9444f324e2f6; no voters:
I20260620 19:21:14.592456 19610 raft_consensus.cc:2781] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:14.592509 19610 raft_consensus.cc:482] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:14.592540 19610 raft_consensus.cc:3037] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:14.593279 19610 raft_consensus.cc:504] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.593421 19610 leader_election.cc:288] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [CANDIDATE]: Term 1 election: Requested vote from peers d740b4dbb36c4030ae8b9444f324e2f6 (127.1.1.194:37507), 32812968a6d84abbbd50fa22f1ff4043 (127.1.1.193:34925)
I20260620 19:21:14.593618 19409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb6bd5dca06469b83a6a612ae076c56" candidate_uuid: "5eedcecaefd849abae52c297b7e8a9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d740b4dbb36c4030ae8b9444f324e2f6"
I20260620 19:21:14.593695 19409 raft_consensus.cc:3037] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:14.593763 19275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3eb6bd5dca06469b83a6a612ae076c56" candidate_uuid: "5eedcecaefd849abae52c297b7e8a9b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32812968a6d84abbbd50fa22f1ff4043"
I20260620 19:21:14.593849 19275 raft_consensus.cc:3037] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:14.594487 19409 raft_consensus.cc:2454] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eedcecaefd849abae52c297b7e8a9b9 in term 1.
I20260620 19:21:14.594617 19275 raft_consensus.cc:2454] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eedcecaefd849abae52c297b7e8a9b9 in term 1.
I20260620 19:21:14.594663 19476 leader_election.cc:302] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eedcecaefd849abae52c297b7e8a9b9, d740b4dbb36c4030ae8b9444f324e2f6; no voters:
I20260620 19:21:14.594738 19610 raft_consensus.cc:2781] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:14.594875 19610 raft_consensus.cc:686] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 1 LEADER]: Becoming Leader. State: Replica: 5eedcecaefd849abae52c297b7e8a9b9, State: Running, Role: LEADER
I20260620 19:21:14.594972 19610 consensus_queue.cc:237] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } }
I20260620 19:21:14.595640 19130 catalog_manager.cc:5495] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eedcecaefd849abae52c297b7e8a9b9 (127.1.1.195). New cstate: current_term: 1 leader_uuid: "5eedcecaefd849abae52c297b7e8a9b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eedcecaefd849abae52c297b7e8a9b9" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 44459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 } health_report { overall_health: UNKNOWN } } }
I20260620 19:21:14.644207 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.642231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48742 (local address 127.1.1.195:44459)
0620 19:21:14.642284 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.642286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.642292 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:21:14.642315 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.642333 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.642348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.642411 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:14.642519 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.642950 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.643627 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.643772 (+ 145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.643836 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.643967 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:14.644038 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:14.644099 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.644121 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.644128 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:21:14.645766 19275 raft_consensus.cc:1261] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 1 FOLLOWER]: Refusing update from remote peer 5eedcecaefd849abae52c297b7e8a9b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:14.645799 19409 raft_consensus.cc:1261] T 3eb6bd5dca06469b83a6a612ae076c56 P d740b4dbb36c4030ae8b9444f324e2f6 [term 1 FOLLOWER]: Refusing update from remote peer 5eedcecaefd849abae52c297b7e8a9b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:14.646034 19610 consensus_queue.cc:1035] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d740b4dbb36c4030ae8b9444f324e2f6" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:14.646183 19613 consensus_queue.cc:1035] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32812968a6d84abbbd50fa22f1ff4043" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:14.648169 19616 mvcc.cc:205] Tried to move back new op lower bound from 7299003492946640896 to 7299003492741341184. Current Snapshot: MvccSnapshot[applied={T|T < 7299003492946640896}]
I20260620 19:21:14.649138 19618 mvcc.cc:205] Tried to move back new op lower bound from 7299003492946640896 to 7299003492741341184. Current Snapshot: MvccSnapshot[applied={T|T < 7299003492946640896}]
I20260620 19:21:14.649780 19617 mvcc.cc:205] Tried to move back new op lower bound from 7299003492946640896 to 7299003492741341184. Current Snapshot: MvccSnapshot[applied={T|T < 7299003492946640896 or (T in {7299003492946640896})}]
I20260620 19:21:14.650694 19130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:14.650784 19130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:14.651496 19130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 429e76bd780d47e7888c394eee5d0b06: Sending DeleteTablet for 3 replicas of tablet 3eb6bd5dca06469b83a6a612ae076c56
I20260620 19:21:14.651877 19389 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:14 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:48688
I20260620 19:21:14.651876 19255 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:14 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:56956
I20260620 19:21:14.652065 19523 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb6bd5dca06469b83a6a612ae076c56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:14 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:48740
I20260620 19:21:14.652084 19625 tablet_replica.cc:330] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: stopping tablet replica
I20260620 19:21:14.652171 19625 raft_consensus.cc:2229] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:14.652240 19627 tablet_replica.cc:330] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: stopping tablet replica
I20260620 19:21:14.652256 19625 raft_consensus.cc:2258] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:14.652364 19627 raft_consensus.cc:2229] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:21:14.652484 19627 raft_consensus.cc:2258] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:14.652596 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19192
I20260620 19:21:14.652779 19625 ts_tablet_manager.cc:1905] T 3eb6bd5dca06469b83a6a612ae076c56 P 32812968a6d84abbbd50fa22f1ff4043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:14.653661 19627 ts_tablet_manager.cc:1905] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:14.655411 19627 ts_tablet_manager.cc:1918] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:14.655498 19627 log.cc:1196] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/3eb6bd5dca06469b83a6a612ae076c56
I20260620 19:21:14.655782 19627 ts_tablet_manager.cc:1939] T 3eb6bd5dca06469b83a6a612ae076c56 P 5eedcecaefd849abae52c297b7e8a9b9: Deleting consensus metadata
I20260620 19:21:14.658941 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19326
I20260620 19:21:14.658959 19117 catalog_manager.cc:4841] TS 5eedcecaefd849abae52c297b7e8a9b9 (127.1.1.195:44459): tablet 3eb6bd5dca06469b83a6a612ae076c56 (table test-table [id=17cd52cf31dc4a6eb22024dd7260adfa]) successfully deleted
W20260620 19:21:14.659062 19117 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:34925 (error 108)
I20260620 19:21:14.659780 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.659676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34925 (local address 127.0.0.1:56960)
0620 19:21:14.659715 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.659743 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:21:14.659965 19117 catalog_manager.cc:4584] TS 32812968a6d84abbbd50fa22f1ff4043 (127.1.1.193:34925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3eb6bd5dca06469b83a6a612ae076c56: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34925: connect: Connection refused (error 111)
W20260620 19:21:14.664554 19599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:14.664460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:37507 (local address 127.0.0.1:48696)
0620 19:21:14.664503 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.664526 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.194:37507: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:21:14.664776 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19460
I20260620 19:21:14.668236 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19098
2026-06-20T19:21:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1910 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:14Z Disabled control of system clock
I20260620 19:21:14.693029 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35801
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:14.765295 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:14.765472 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:14.765489 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:14.766444 19632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:14.766479 19632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:14.766489 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:14.766499 19632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:14.766510 19632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:14.767588 19632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:35801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:14.767760 19632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:14.767961 19632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:14.772008 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.772053 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.772051 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.772202 19632 server_base.cc:1034] running on GCE node
I20260620 19:21:14.772423 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:14.772709 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:14.773854 19632 hybrid_clock.cc:648] HybridClock initialized: now 1781983274773846 us; error 26 us; skew 500 ppm
I20260620 19:21:14.774892 19632 webserver.cc:460] Webserver started at http://127.1.1.254:46691/ using document root <none> and password file <none>
I20260620 19:21:14.775101 19632 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:14.775161 19632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:14.775288 19632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:14.776160 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "2add0646f9f34add8cee22b12ef3d123"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "1f863d74d5970d079ba1f846d95c81e7"
server_key_iv: "824084ebb1f6de03832161fd10d0378a"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.776446 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "2add0646f9f34add8cee22b12ef3d123"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "1f863d74d5970d079ba1f846d95c81e7"
server_key_iv: "824084ebb1f6de03832161fd10d0378a"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.777560 19632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:14.778184 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.778373 19632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:14.778440 19632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2add0646f9f34add8cee22b12ef3d123"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "1f863d74d5970d079ba1f846d95c81e7"
server_key_iv: "824084ebb1f6de03832161fd10d0378a"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.778489 19632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:14.796674 19632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:14.796866 19632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:14.796947 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:14.800302 19632 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:35801
I20260620 19:21:14.800329 19698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:35801 every 8 connection(s)
I20260620 19:21:14.800612 19632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:14.801393 19699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:14.803135 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: Bootstrap starting.
I20260620 19:21:14.803592 19699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:14.803788 19699 log.cc:824] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:14.804280 19699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: No bootstrap required, opened a new log
I20260620 19:21:14.805662 19699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } }
I20260620 19:21:14.805768 19699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:14.805786 19699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2add0646f9f34add8cee22b12ef3d123, State: Initialized, Role: FOLLOWER
I20260620 19:21:14.805891 19699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } }
I20260620 19:21:14.805962 19699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:14.805987 19699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:14.806016 19699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:14.806418 19699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } }
I20260620 19:21:14.806429 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19632
I20260620 19:21:14.806489 19699 leader_election.cc:302] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2add0646f9f34add8cee22b12ef3d123; no voters:
I20260620 19:21:14.806525 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:14.806650 19699 leader_election.cc:288] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:14.806681 19702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:14.806711 1031 external_mini_cluster.cc:1428] Setting key 35ac175effbd272db18bd26cf376abcd
I20260620 19:21:14.806805 19702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [term 1 LEADER]: Becoming Leader. State: Replica: 2add0646f9f34add8cee22b12ef3d123, State: Running, Role: LEADER
I20260620 19:21:14.806882 19699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:14.806876 19702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } }
I20260620 19:21:14.807350 19702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2add0646f9f34add8cee22b12ef3d123. Latest consensus state: current_term: 1 leader_uuid: "2add0646f9f34add8cee22b12ef3d123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } } }
I20260620 19:21:14.807585 19702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:14.807521 19704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2add0646f9f34add8cee22b12ef3d123" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2add0646f9f34add8cee22b12ef3d123" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35801 } } }
I20260620 19:21:14.807893 19704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:14.808117 19713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:14.808552 19713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:14.809913 19713 catalog_manager.cc:1347] Generated new cluster ID: 0716d2c4ebca4329a4c25ca636799c91
I20260620 19:21:14.809969 19713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:14.811827 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.807147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39824 (local address 127.1.1.254:35801)
0620 19:21:14.807414 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.807422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.807454 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:21:14.807575 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.807582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.807652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.808011 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.808109 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.809120 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.810110 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.810618 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.810712 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.811272 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.811276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.811276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:14.811432 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.811463 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:14.811474 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.811527 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.811609 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.811668 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":51,"threads_started":1}
I20260620 19:21:14.826328 19713 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:14.826779 19713 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:14.829908 19713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: Generated new TSK 0
I20260620 19:21:14.830070 19713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:14.832060 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--builtin_ntp_servers=127.1.1.212:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:14.905804 19723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:14.905985 19723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:14.906006 19723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:14.907166 19723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:14.907200 19723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:14.907303 19723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:14.908710 19723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:14.908921 19723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:14.909278 19723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:14.914669 19728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:14.914700 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.914875 19723 server_base.cc:1034] running on GCE node
W20260620 19:21:14.914644 19731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:14.915089 19723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:14.915354 19723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:14.916517 19723 hybrid_clock.cc:648] HybridClock initialized: now 1781983274916504 us; error 30 us; skew 500 ppm
I20260620 19:21:14.922443 19723 webserver.cc:460] Webserver started at http://127.1.1.193:40597/ using document root <none> and password file <none>
I20260620 19:21:14.922663 19723 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:14.922729 19723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:14.922874 19723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:14.923840 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "857420cddec94a27a06016803ec1ac16"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "d2aa955ca46df1e16df85ea0d6e71496"
server_key_iv: "f9956b8dc9a117605657d5269c79fb26"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.924180 19723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "857420cddec94a27a06016803ec1ac16"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "d2aa955ca46df1e16df85ea0d6e71496"
server_key_iv: "f9956b8dc9a117605657d5269c79fb26"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.925499 19723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:14.926275 19737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.926448 19723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:14.926539 19723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "857420cddec94a27a06016803ec1ac16"
format_stamp: "Formatted at 2026-06-20 19:21:14 on dist-test-slave-mwkb"
server_key: "d2aa955ca46df1e16df85ea0d6e71496"
server_key_iv: "f9956b8dc9a117605657d5269c79fb26"
server_key_version: "encryptionkey@0"
I20260620 19:21:14.926617 19723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:14.944736 19723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:14.945004 19723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:14.945151 19723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:14.945345 19723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:14.945607 19723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:14.945655 19723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.945691 19723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:14.945716 19723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:14.950960 19723 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:45377
I20260620 19:21:14.951035 19850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:45377 every 8 connection(s)
I20260620 19:21:14.951287 19723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:21:14.955719 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.951932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:49989 (local address 127.1.1.254:35801)
0620 19:21:14.952004 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:14.952006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:14.952191 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:21:14.952611 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:14.952612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:14.952622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:14.952672 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:14.953327 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.953777 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.954634 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.954816 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.954864 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.955507 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:14.955508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:14.955509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:14.955513 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:14.955525 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:14.955529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:14.955580 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:14.955648 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:21:14.955670 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:21:14.955849 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:14.951902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35801 (local address 127.1.1.193:49989)
0620 19:21:14.952069 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:14.952096 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:14.952537 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:14.952719 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:14.952721 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:14.952729 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:14.953254 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.953259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.953852 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.953852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.954563 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:14.954565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:14.954888 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:14.954889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:14.954993 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:14.955389 (+ 396us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:14.955390 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:14.955414 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:14.955432 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:14.955590 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:14.955594 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:14.955596 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:14.955638 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:21:14.955697 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260620 19:21:14.956243 19851 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35801
I20260620 19:21:14.956329 19851 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:14.956532 19851 heartbeater.cc:511] Master 127.1.1.254:35801 requested a full tablet report, sending...
I20260620 19:21:14.956629 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19723
I20260620 19:21:14.956720 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:21:14.956876 1031 external_mini_cluster.cc:1428] Setting key f880bf768e47dbcb47d2748afccd3ebc
I20260620 19:21:14.957046 19663 ts_manager.cc:194] Registered new tserver with Master: 857420cddec94a27a06016803ec1ac16 (127.1.1.193:45377)
I20260620 19:21:14.957840 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--builtin_ntp_servers=127.1.1.212:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:21:14.957970 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.193:49989
W20260620 19:21:15.029230 19854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:15.029418 19854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:15.029451 19854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:15.030511 19854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:15.030570 19854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:15.030611 19854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:15.031782 19854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:15.032012 19854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:15.032253 19854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:15.036890 19859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:15.036921 19862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:15.037016 19860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:15.036926 19854 server_base.cc:1034] running on GCE node
I20260620 19:21:15.037214 19854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:15.037446 19854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:15.038615 19854 hybrid_clock.cc:648] HybridClock initialized: now 1781983275038601 us; error 22 us; skew 500 ppm
I20260620 19:21:15.039700 19854 webserver.cc:460] Webserver started at http://127.1.1.194:42243/ using document root <none> and password file <none>
I20260620 19:21:15.039921 19854 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:15.039987 19854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:15.040113 19854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:15.041154 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "424525f77f7d491a84a38964a5afebd4"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "1c5c95a7bf30caa8230e100c50c192b2"
server_key_iv: "132d2823ae005261a2afed8db6a1f1fc"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.041558 19854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "424525f77f7d491a84a38964a5afebd4"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "1c5c95a7bf30caa8230e100c50c192b2"
server_key_iv: "132d2823ae005261a2afed8db6a1f1fc"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.042690 19854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:21:15.043354 19868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.043550 19854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:15.043646 19854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "424525f77f7d491a84a38964a5afebd4"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "1c5c95a7bf30caa8230e100c50c192b2"
server_key_iv: "132d2823ae005261a2afed8db6a1f1fc"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.043720 19854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:15.061303 19854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:15.061520 19854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:15.061643 19854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:15.061846 19854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:15.062130 19854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:15.062180 19854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.062215 19854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:15.062239 19854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.067411 19854 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:35343
I20260620 19:21:15.067456 19981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:35343 every 8 connection(s)
I20260620 19:21:15.067691 19854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:21:15.071557 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.068258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:60107 (local address 127.1.1.254:35801)
0620 19:21:15.068298 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.068300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.068477 (+ 177us) server_negotiation.cc:407] Connection header received
0620 19:21:15.068797 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.068798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.068806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.068852 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:15.069348 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.069751 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.070546 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.070705 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.070749 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.071323 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:15.071324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:15.071324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:15.071330 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:15.071345 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:15.071349 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:15.071417 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.071438 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.071461 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:15.071640 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.068222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35801 (local address 127.1.1.194:60107)
0620 19:21:15.068370 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.068401 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.068730 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.068890 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.068893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.068900 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:15.069285 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.069288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.069815 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.069816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.070484 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.070485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.070768 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.070769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.070855 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.071198 (+ 343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:15.071199 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:15.071220 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:15.071236 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:15.071401 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:15.071403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:15.071405 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.071438 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.071495 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260620 19:21:15.071784 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19854
I20260620 19:21:15.071887 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:15.072072 1031 external_mini_cluster.cc:1428] Setting key 3676bf8d951ae08209243a267aebb898
I20260620 19:21:15.072057 19982 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35801
I20260620 19:21:15.072149 19982 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:15.072311 19982 heartbeater.cc:511] Master 127.1.1.254:35801 requested a full tablet report, sending...
I20260620 19:21:15.072631 19663 ts_manager.cc:194] Registered new tserver with Master: 424525f77f7d491a84a38964a5afebd4 (127.1.1.194:35343)
I20260620 19:21:15.072990 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.194:60107
I20260620 19:21:15.073406 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--builtin_ntp_servers=127.1.1.212:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:15.144575 19985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:15.144786 19985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:15.144841 19985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:15.145978 19985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:15.146030 19985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:15.146080 19985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:15.147258 19985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:15.147465 19985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:15.147698 19985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:15.151870 19991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:15.151870 19993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:15.151959 19990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:15.152027 19985 server_base.cc:1034] running on GCE node
I20260620 19:21:15.152261 19985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:15.152472 19985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:15.153617 19985 hybrid_clock.cc:648] HybridClock initialized: now 1781983275153605 us; error 26 us; skew 500 ppm
I20260620 19:21:15.154696 19985 webserver.cc:460] Webserver started at http://127.1.1.195:45189/ using document root <none> and password file <none>
I20260620 19:21:15.154913 19985 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:15.154958 19985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:15.155115 19985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:15.156031 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "61e167ba370d44298773941e4d6236ec"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "25c896e7e33b6b9fdcf4d78dedc5a517"
server_key_iv: "df23b97c93e4abef652bfc4a3ebc38c5"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.156337 19985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "61e167ba370d44298773941e4d6236ec"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "25c896e7e33b6b9fdcf4d78dedc5a517"
server_key_iv: "df23b97c93e4abef652bfc4a3ebc38c5"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.157462 19985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:15.158053 19999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.158222 19985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:15.158284 19985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "61e167ba370d44298773941e4d6236ec"
format_stamp: "Formatted at 2026-06-20 19:21:15 on dist-test-slave-mwkb"
server_key: "25c896e7e33b6b9fdcf4d78dedc5a517"
server_key_iv: "df23b97c93e4abef652bfc4a3ebc38c5"
server_key_version: "encryptionkey@0"
I20260620 19:21:15.158390 19985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:15.183066 19985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:15.183307 19985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:15.183449 19985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:15.183658 19985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:15.183966 19985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:15.183995 19985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.184012 19985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:15.184062 19985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:15.189044 19985 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:39997
I20260620 19:21:15.189111 20112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:39997 every 8 connection(s)
I20260620 19:21:15.189356 19985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:21:15.193341 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.189923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:50697 (local address 127.1.1.254:35801)
0620 19:21:15.189986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.189988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.190130 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:21:15.190486 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.190486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.190494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.190536 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:15.191061 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.191482 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.192223 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.192396 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.192444 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.193128 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:15.193130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:15.193131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:15.193135 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:15.193151 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:15.193156 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:15.193203 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.193264 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.193286 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:21:15.193548 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.189891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:35801 (local address 127.1.1.195:50697)
0620 19:21:15.190042 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.190073 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.190419 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.190601 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.190603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.190610 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:15.191004 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.191007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.191550 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.191550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.192161 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.192163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.192491 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.192492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.192588 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.192979 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:15.192979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:15.193002 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:15.193020 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:15.193203 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:15.193206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:15.193208 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.193254 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.193320 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":83,"threads_started":1}
I20260620 19:21:15.193946 20113 heartbeater.cc:347] Connected to a master server at 127.1.1.254:35801
I20260620 19:21:15.194022 20113 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:15.194207 20113 heartbeater.cc:511] Master 127.1.1.254:35801 requested a full tablet report, sending...
I20260620 19:21:15.194522 19663 ts_manager.cc:194] Registered new tserver with Master: 61e167ba370d44298773941e4d6236ec (127.1.1.195:39997)
I20260620 19:21:15.194862 19663 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.195:50697
I20260620 19:21:15.197630 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 19985
I20260620 19:21:15.197700 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:15.197817 1031 external_mini_cluster.cc:1428] Setting key 0fe2bccdc91141b5f6defda7c7ef8f3d
I20260620 19:21:15.198565 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:15.202286 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.199701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.1.1.254:35801)
0620 19:21:15.199772 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.199773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.199874 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:21:15.199964 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.199965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.199976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.200037 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:15.200200 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.200623 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.201346 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.201503 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.201572 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.202052 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:15.202053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:15.202054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:15.202057 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:15.202067 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:15.202070 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:15.202161 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.202217 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.202224 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:15.203675 19663 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:15.204965 19663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:15.210736 20121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.206881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:35343 (local address 127.0.0.1:38270)
0620 19:21:15.206989 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.207024 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.207089 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.207466 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.207468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.207470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:15.207510 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.207511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208766 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.208767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.209681 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.209683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210472 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.210535 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.210546 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.210616 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.210657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":75,"thread_start_us":59,"threads_started":1}
I20260620 19:21:15.210745 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.207024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.1.1.194:35343)
0620 19:21:15.207285 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.207289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.207297 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:15.207353 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.207355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.207369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.207458 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:15.207629 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208676 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.209763 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210392 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210455 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.210524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.210629 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.210667 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":86,"threads_started":1}
I20260620 19:21:15.210942 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.206922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:45377 (local address 127.0.0.1:33390)
0620 19:21:15.207042 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.207051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.207095 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.207563 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.207564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.207566 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:15.207598 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.207599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208809 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.208809 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.209992 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.209994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210717 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.210774 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.210779 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.210822 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.210876 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":62,"thread_start_us":63,"threads_started":1}
I20260620 19:21:15.211089 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.207235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.1.1.195:39997)
0620 19:21:15.207657 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.207660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.207667 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:15.207689 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.207691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.207701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.207945 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:15.208097 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208681 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210167 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210682 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210733 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.210799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.210970 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.211011 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":81,"threads_started":1}
I20260620 19:21:15.211133 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.207083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:39997 (local address 127.0.0.1:44356)
0620 19:21:15.207544 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.207553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.207591 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.207986 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.207987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.207989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:15.208027 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.208028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208949 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.208949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.210096 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.210097 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210846 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.210907 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.210911 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.210957 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.211064 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260620 19:21:15.211282 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.207216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.1.1.193:45377)
0620 19:21:15.207395 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.207398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.207407 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:15.207454 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.207455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.207468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.207551 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:15.207683 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.208725 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.210052 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.210635 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.211056 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.211141 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.211167 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.211213 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260620 19:21:15.211756 20047 tablet_service.cc:1467] Processing CreateTablet for tablet b55bff4f711146d0a95910dab52279f6 (DEFAULT_TABLE table=test-table [id=9c956ae33d4f4129b35d5122a7cd3b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:15.211833 19785 tablet_service.cc:1467] Processing CreateTablet for tablet b55bff4f711146d0a95910dab52279f6 (DEFAULT_TABLE table=test-table [id=9c956ae33d4f4129b35d5122a7cd3b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:15.211848 19916 tablet_service.cc:1467] Processing CreateTablet for tablet b55bff4f711146d0a95910dab52279f6 (DEFAULT_TABLE table=test-table [id=9c956ae33d4f4129b35d5122a7cd3b61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:15.212065 20047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55bff4f711146d0a95910dab52279f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:15.212110 19916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55bff4f711146d0a95910dab52279f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:15.212119 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55bff4f711146d0a95910dab52279f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:15.214601 20127 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: Bootstrap starting.
I20260620 19:21:15.215152 20129 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Bootstrap starting.
I20260620 19:21:15.215152 20128 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Bootstrap starting.
I20260620 19:21:15.215463 20127 tablet_bootstrap.cc:654] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:15.215723 20127 log.cc:824] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:15.215749 20128 tablet_bootstrap.cc:654] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:15.215749 20129 tablet_bootstrap.cc:654] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:15.216038 20129 log.cc:824] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:15.216030 20128 log.cc:824] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:15.216470 20127 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: No bootstrap required, opened a new log
I20260620 19:21:15.216548 20127 ts_tablet_manager.cc:1397] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:15.216722 20129 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: No bootstrap required, opened a new log
I20260620 19:21:15.216800 20129 ts_tablet_manager.cc:1397] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:15.216902 20128 tablet_bootstrap.cc:492] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: No bootstrap required, opened a new log
I20260620 19:21:15.216960 20128 ts_tablet_manager.cc:1397] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:15.218276 20127 raft_consensus.cc:348] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.218568 20127 raft_consensus.cc:374] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:15.218611 20127 raft_consensus.cc:729] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857420cddec94a27a06016803ec1ac16, State: Initialized, Role: FOLLOWER
I20260620 19:21:15.218714 20127 consensus_queue.cc:260] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.218966 20127 ts_tablet_manager.cc:1428] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:21:15.219094 19851 heartbeater.cc:503] Master 127.1.1.254:35801 was elected leader, sending a full tablet report...
I20260620 19:21:15.219132 20129 raft_consensus.cc:348] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.219185 20128 raft_consensus.cc:348] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.219278 20129 raft_consensus.cc:374] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:15.219327 20128 raft_consensus.cc:374] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:15.219327 20129 raft_consensus.cc:729] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e167ba370d44298773941e4d6236ec, State: Initialized, Role: FOLLOWER
I20260620 19:21:15.219367 20128 raft_consensus.cc:729] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 424525f77f7d491a84a38964a5afebd4, State: Initialized, Role: FOLLOWER
I20260620 19:21:15.219451 20129 consensus_queue.cc:260] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.219471 20128 consensus_queue.cc:260] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.219695 20129 ts_tablet_manager.cc:1428] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:21:15.219735 20113 heartbeater.cc:503] Master 127.1.1.254:35801 was elected leader, sending a full tablet report...
I20260620 19:21:15.219695 20128 ts_tablet_manager.cc:1428] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:21:15.219915 19982 heartbeater.cc:503] Master 127.1.1.254:35801 was elected leader, sending a full tablet report...
W20260620 19:21:15.318147 19983 tablet.cc:2369] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:15.323225 20135 raft_consensus.cc:482] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:15.323298 20135 raft_consensus.cc:504] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.323650 20135 leader_election.cc:288] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61e167ba370d44298773941e4d6236ec (127.1.1.195:39997), 857420cddec94a27a06016803ec1ac16 (127.1.1.193:45377)
I20260620 19:21:15.326133 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.323866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:46029 (local address 127.1.1.193:45377)
0620 19:21:15.323900 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.323902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.323910 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:15.323966 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.323967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.323976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.324057 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:15.324184 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.324580 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325497 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.325819 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325900 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.325910 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.326044 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.326073 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:21:15.326169 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.323793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:45377 (local address 127.1.1.194:46029)
0620 19:21:15.323847 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.323859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.323906 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.324061 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.324062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.324063 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:15.324120 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.324121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.324650 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.324651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.325419 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.325421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.325902 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.325976 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.325978 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.326029 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.326067 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":20}
I20260620 19:21:15.326161 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.323858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:55407 (local address 127.1.1.195:39997)
0620 19:21:15.323897 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.323899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.324008 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:21:15.324083 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.324084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.324097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.324148 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:15.324280 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.324658 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325530 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.325845 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325921 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.325930 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.326072 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.326104 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:21:15.326187 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.323793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:39997 (local address 127.1.1.194:55407)
0620 19:21:15.323929 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.323937 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.324028 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:15.324152 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:15.324153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:15.324155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:15.324213 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.324215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.324730 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.324731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.325431 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:15.325433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.325923 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.325924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:15.326001 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.326006 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:21:15.326068 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:21:15.326101 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":11,"thread_start_us":73,"threads_started":1}
I20260620 19:21:15.326414 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55bff4f711146d0a95910dab52279f6" candidate_uuid: "424525f77f7d491a84a38964a5afebd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e167ba370d44298773941e4d6236ec" is_pre_election: true
I20260620 19:21:15.326512 20067 raft_consensus.cc:2454] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424525f77f7d491a84a38964a5afebd4 in term 0.
I20260620 19:21:15.326674 19805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55bff4f711146d0a95910dab52279f6" candidate_uuid: "424525f77f7d491a84a38964a5afebd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857420cddec94a27a06016803ec1ac16" is_pre_election: true
I20260620 19:21:15.326709 19870 leader_election.cc:302] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424525f77f7d491a84a38964a5afebd4, 61e167ba370d44298773941e4d6236ec; no voters:
I20260620 19:21:15.326772 19805 raft_consensus.cc:2454] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 424525f77f7d491a84a38964a5afebd4 in term 0.
I20260620 19:21:15.326841 20135 raft_consensus.cc:2781] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:15.326895 20135 raft_consensus.cc:482] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:15.326920 20135 raft_consensus.cc:3037] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:15.327493 20135 raft_consensus.cc:504] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.327610 20135 leader_election.cc:288] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [CANDIDATE]: Term 1 election: Requested vote from peers 61e167ba370d44298773941e4d6236ec (127.1.1.195:39997), 857420cddec94a27a06016803ec1ac16 (127.1.1.193:45377)
I20260620 19:21:15.327714 20067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55bff4f711146d0a95910dab52279f6" candidate_uuid: "424525f77f7d491a84a38964a5afebd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61e167ba370d44298773941e4d6236ec"
I20260620 19:21:15.327760 19805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55bff4f711146d0a95910dab52279f6" candidate_uuid: "424525f77f7d491a84a38964a5afebd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "857420cddec94a27a06016803ec1ac16"
I20260620 19:21:15.327780 20067 raft_consensus.cc:3037] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:15.327809 19805 raft_consensus.cc:3037] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:15.328538 20067 raft_consensus.cc:2454] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424525f77f7d491a84a38964a5afebd4 in term 1.
I20260620 19:21:15.328538 19805 raft_consensus.cc:2454] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 424525f77f7d491a84a38964a5afebd4 in term 1.
I20260620 19:21:15.328687 19869 leader_election.cc:302] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 424525f77f7d491a84a38964a5afebd4, 857420cddec94a27a06016803ec1ac16; no voters:
I20260620 19:21:15.328783 20135 raft_consensus.cc:2781] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:15.328913 20135 raft_consensus.cc:686] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 1 LEADER]: Becoming Leader. State: Replica: 424525f77f7d491a84a38964a5afebd4, State: Running, Role: LEADER
I20260620 19:21:15.329020 20135 consensus_queue.cc:237] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } }
I20260620 19:21:15.329732 19662 catalog_manager.cc:5495] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 424525f77f7d491a84a38964a5afebd4 (127.1.1.194). New cstate: current_term: 1 leader_uuid: "424525f77f7d491a84a38964a5afebd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "424525f77f7d491a84a38964a5afebd4" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 } health_report { overall_health: UNKNOWN } } }
I20260620 19:21:15.353528 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:15.351623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.1.1.194:35343)
0620 19:21:15.351671 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:15.351672 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:15.351678 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:21:15.351718 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:15.351718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:15.351731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:15.351778 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:15.351871 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.352222 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.352871 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:15.353035 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:15.353127 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:15.353244 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:15.353335 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:15.353403 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:15.353464 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:21:15.353471 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:21:15.355011 19805 raft_consensus.cc:1261] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 1 FOLLOWER]: Refusing update from remote peer 424525f77f7d491a84a38964a5afebd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:15.355011 20067 raft_consensus.cc:1261] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 1 FOLLOWER]: Refusing update from remote peer 424525f77f7d491a84a38964a5afebd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:15.355321 20137 consensus_queue.cc:1035] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e167ba370d44298773941e4d6236ec" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:15.355460 20135 consensus_queue.cc:1035] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "857420cddec94a27a06016803ec1ac16" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 45377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:15.357376 20140 mvcc.cc:205] Tried to move back new op lower bound from 7299003495851753472 to 7299003495748190208. Current Snapshot: MvccSnapshot[applied={T|T < 7299003495851753472}]
I20260620 19:21:15.357892 20141 mvcc.cc:205] Tried to move back new op lower bound from 7299003495851753472 to 7299003495748190208. Current Snapshot: MvccSnapshot[applied={T|T < 7299003495851753472}]
I20260620 19:21:15.358026 20142 mvcc.cc:205] Tried to move back new op lower bound from 7299003495851753472 to 7299003495748190208. Current Snapshot: MvccSnapshot[applied={T|T < 7299003495851753472 or (T in {7299003495851753472})}]
I20260620 19:21:15.359731 19662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:15.359820 19662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:15.360482 19662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2add0646f9f34add8cee22b12ef3d123: Sending DeleteTablet for 3 replicas of tablet b55bff4f711146d0a95910dab52279f6
I20260620 19:21:15.360785 19785 tablet_service.cc:1514] Processing DeleteTablet for tablet b55bff4f711146d0a95910dab52279f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:15 UTC) from {username='slave'} at 127.0.0.1:33390
I20260620 19:21:15.360785 19916 tablet_service.cc:1514] Processing DeleteTablet for tablet b55bff4f711146d0a95910dab52279f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:15 UTC) from {username='slave'} at 127.0.0.1:38270
I20260620 19:21:15.360966 20150 tablet_replica.cc:330] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16: stopping tablet replica
I20260620 19:21:15.361060 20150 raft_consensus.cc:2229] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:15.361104 20047 tablet_service.cc:1514] Processing DeleteTablet for tablet b55bff4f711146d0a95910dab52279f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:15 UTC) from {username='slave'} at 127.0.0.1:44356
I20260620 19:21:15.361176 20150 raft_consensus.cc:2258] T b55bff4f711146d0a95910dab52279f6 P 857420cddec94a27a06016803ec1ac16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:15.361280 20151 tablet_replica.cc:330] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: stopping tablet replica
I20260620 19:21:15.361290 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19723
I20260620 19:21:15.361755 20151 raft_consensus.cc:2229] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:21:15.361893 20151 raft_consensus.cc:2258] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:15.362458 20151 ts_tablet_manager.cc:1905] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:15.364315 20151 ts_tablet_manager.cc:1918] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:15.364382 20151 log.cc:1196] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/b55bff4f711146d0a95910dab52279f6
I20260620 19:21:15.364601 20151 ts_tablet_manager.cc:1939] T b55bff4f711146d0a95910dab52279f6 P 424525f77f7d491a84a38964a5afebd4: Deleting consensus metadata
I20260620 19:21:15.364977 19648 catalog_manager.cc:4841] TS 424525f77f7d491a84a38964a5afebd4 (127.1.1.194:35343): tablet b55bff4f711146d0a95910dab52279f6 (table test-table [id=9c956ae33d4f4129b35d5122a7cd3b61]) successfully deleted
W20260620 19:21:15.367678 19647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:45377 (error 108)
I20260620 19:21:15.367820 20152 tablet_replica.cc:330] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: stopping tablet replica
I20260620 19:21:15.367945 20152 raft_consensus.cc:2229] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:15.368068 20152 raft_consensus.cc:2258] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:21:15.368083 20124 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:15.367949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:45377 (local address 127.0.0.1:33392)
0620 19:21:15.368002 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:15.368011 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:15.368043 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:45377: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 19:21:15.368173 19647 catalog_manager.cc:4584] TS 857420cddec94a27a06016803ec1ac16 (127.1.1.193:45377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b55bff4f711146d0a95910dab52279f6: Network error: Client connection negotiation failed: client connection to 127.1.1.193:45377: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260620 19:21:15.368283 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19854
I20260620 19:21:15.368844 20152 ts_tablet_manager.cc:1905] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:15.370319 20152 ts_tablet_manager.cc:1918] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:15.370393 20152 log.cc:1196] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/b55bff4f711146d0a95910dab52279f6
I20260620 19:21:15.370581 20152 ts_tablet_manager.cc:1939] T b55bff4f711146d0a95910dab52279f6 P 61e167ba370d44298773941e4d6236ec: Deleting consensus metadata
I20260620 19:21:15.370914 19648 catalog_manager.cc:4841] TS 61e167ba370d44298773941e4d6236ec (127.1.1.195:39997): tablet b55bff4f711146d0a95910dab52279f6 (table test-table [id=9c956ae33d4f4129b35d5122a7cd3b61]) successfully deleted
I20260620 19:21:15.372949 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19985
I20260620 19:21:15.376955 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 19632
2026-06-20T19:21:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (708 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:15 dist-test-slave-mwkb krb5kdc[20156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:15 dist-test-slave-mwkb krb5kdc[20156](info): set up 2 sockets
Jun 20 19:21:15 dist-test-slave-mwkb krb5kdc[20156](info): commencing operation
krb5kdc: starting...
W20260620 19:21:16.316515 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:16.415495 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38645
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:16.485673 20169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:16.485826 20169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:16.485846 20169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:16.486816 20169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:16.486846 20169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:16.486860 20169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:16.486871 20169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:16.486881 20169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:16.488090 20169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:16.488238 20169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:16.488370 20169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:16.492520 20177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.492506 20174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.492640 20175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:16.492863 20169 server_base.cc:1034] running on GCE node
I20260620 19:21:16.493144 20169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:16.493412 20169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:16.494571 20169 hybrid_clock.cc:648] HybridClock initialized: now 1781983276494556 us; error 31 us; skew 500 ppm
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:16.496738 20169 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:16.497347 20169 webserver.cc:460] Webserver started at http://127.1.1.254:39901/ using document root <none> and password file <none>
I20260620 19:21:16.497517 20169 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:16.497555 20169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:16.497645 20169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:16.498536 20169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "c250adb92d0e40ec973346c62c1cfa7c"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "3442c8ec4d96e47d85da2cbce6c0ceb4"
server_key_iv: "c712cb49451bad0e01de44ac65c5cbeb"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.498853 20169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "c250adb92d0e40ec973346c62c1cfa7c"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "3442c8ec4d96e47d85da2cbce6c0ceb4"
server_key_iv: "c712cb49451bad0e01de44ac65c5cbeb"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.499994 20169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:16.500612 20184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.500748 20169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:16.500819 20169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "c250adb92d0e40ec973346c62c1cfa7c"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "3442c8ec4d96e47d85da2cbce6c0ceb4"
server_key_iv: "c712cb49451bad0e01de44ac65c5cbeb"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.500866 20169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:16.514528 20169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:16.515950 20169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:16.516040 20169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:16.519539 20169 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:38645
I20260620 19:21:16.519589 20236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:38645 every 8 connection(s)
I20260620 19:21:16.519877 20169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:16.520639 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:16.522527 20237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: Bootstrap starting.
I20260620 19:21:16.523073 20237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:16.523351 20237 log.cc:824] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:16.523980 20237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: No bootstrap required, opened a new log
I20260620 19:21:16.525357 20237 raft_consensus.cc:348] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } }
I20260620 19:21:16.525485 20237 raft_consensus.cc:374] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:16.525519 20237 raft_consensus.cc:729] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c250adb92d0e40ec973346c62c1cfa7c, State: Initialized, Role: FOLLOWER
I20260620 19:21:16.525653 20237 consensus_queue.cc:260] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } }
I20260620 19:21:16.525732 20237 raft_consensus.cc:388] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:16.525764 20237 raft_consensus.cc:482] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:16.525806 20237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:16.526347 20237 raft_consensus.cc:504] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } }
I20260620 19:21:16.526455 20237 leader_election.cc:302] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c250adb92d0e40ec973346c62c1cfa7c; no voters:
I20260620 19:21:16.526630 20237 leader_election.cc:288] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:16.526664 20240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:16.526818 20240 raft_consensus.cc:686] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [term 1 LEADER]: Becoming Leader. State: Replica: c250adb92d0e40ec973346c62c1cfa7c, State: Running, Role: LEADER
I20260620 19:21:16.526866 20237 sys_catalog.cc:564] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:16.526898 20240 consensus_queue.cc:237] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } }
I20260620 19:21:16.527251 20241 sys_catalog.cc:455] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c250adb92d0e40ec973346c62c1cfa7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } } }
I20260620 19:21:16.527335 20241 sys_catalog.cc:458] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:16.527347 20242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c250adb92d0e40ec973346c62c1cfa7c. Latest consensus state: current_term: 1 leader_uuid: "c250adb92d0e40ec973346c62c1cfa7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c250adb92d0e40ec973346c62c1cfa7c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 38645 } } }
I20260620 19:21:16.527395 20242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:16.527593 20248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:16.528091 20248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:16.529127 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20169
I20260620 19:21:16.529241 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:16.529435 1031 external_mini_cluster.cc:1428] Setting key 1e68e2c667bcce57aff00696cceae49e
I20260620 19:21:16.529677 20248 catalog_manager.cc:1347] Generated new cluster ID: 59325195d6134bd4817d7f0ca22aeb10
I20260620 19:21:16.529718 20248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:16.539809 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.529954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.1.1.254:38645)
0620 19:21:16.530094 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.530102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.530129 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:21:16.530234 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.530240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.530285 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.530422 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:16.531064 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.532387 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.533410 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.534015 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.534126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.536633 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:16.536637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:16.536637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:16.536647 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:16.538324 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.538684 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.538686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.538687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.538772 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.538959 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.538960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.538960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.539194 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:16.539342 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.539468 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.539606 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":85,"threads_started":1}
I20260620 19:21:16.544559 20248 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:16.545162 20248 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:16.550037 20248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: Generated new TSK 0
I20260620 19:21:16.550172 20248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.1.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
Entry for principal kudu/127.1.1.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab.
I20260620 19:21:16.573359 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:16.646989 20263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:16.647145 20263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:16.647164 20263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:16.648082 20263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:16.648113 20263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:16.648134 20263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:16.649430 20263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.193.keytab
--principal=kudu/127.1.1.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:16.649593 20263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:16.649741 20263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:16.653812 20268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.653812 20269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.653828 20271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:16.654088 20263 server_base.cc:1034] running on GCE node
I20260620 19:21:16.654222 20263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:16.654428 20263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:16.655575 20263 hybrid_clock.cc:648] HybridClock initialized: now 1781983276655574 us; error 40 us; skew 500 ppm
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:16.657713 20263 init.cc:377] Logged in from keytab as kudu/127.1.1.193@KRBTEST.COM (short username kudu)
I20260620 19:21:16.658077 20263 webserver.cc:460] Webserver started at http://127.1.1.193:41101/ using document root <none> and password file <none>
I20260620 19:21:16.658267 20263 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:16.658325 20263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:16.658468 20263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:16.659384 20263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "4fff832381ed478aad0c631cf3a13337"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "4996eeed0ead28d055c798dc5d920565"
server_key_iv: "3d78d6742db2e1fd1c28f15631e621dc"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.659718 20263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "4fff832381ed478aad0c631cf3a13337"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "4996eeed0ead28d055c798dc5d920565"
server_key_iv: "3d78d6742db2e1fd1c28f15631e621dc"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.660881 20263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:16.661528 20278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.661680 20263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.661760 20263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "4fff832381ed478aad0c631cf3a13337"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "4996eeed0ead28d055c798dc5d920565"
server_key_iv: "3d78d6742db2e1fd1c28f15631e621dc"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.661834 20263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:16.680596 20263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:16.682142 20263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:16.682274 20263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:16.682447 20263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:16.682757 20263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:16.682809 20263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.682842 20263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:16.682870 20263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.688289 20263 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:43939
I20260620 19:21:16.688362 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:43939 every 8 connection(s)
I20260620 19:21:16.688632 20263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.193@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:16.696484 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.689278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:59095 (local address 127.1.1.254:38645)
0620 19:21:16.689354 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.689356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.689532 (+ 176us) server_negotiation.cc:407] Connection header received
0620 19:21:16.689919 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.689920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.689930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.689993 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:16.690788 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.691247 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.692130 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.692328 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.692410 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.694305 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:16.694307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:16.694307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:16.694313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:16.695457 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.695781 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.695781 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.695782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.695827 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.696051 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.696052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.696052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.696134 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:16.696177 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.696373 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.696428 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:21:16.696606 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.689235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38645 (local address 127.1.1.193:59095)
0620 19:21:16.689408 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.689453 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.689849 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.690043 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.690045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.690539 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:16.690711 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.690715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.691328 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.691328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.692050 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.692053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.692422 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.692423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.692520 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.692897 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:16.692905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:16.694226 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:16.695523 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.695529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.695531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.695722 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.695904 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.695909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.695909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.696001 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.696182 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:16.696186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:16.696277 (+ 91us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.696347 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.696442 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260620 19:21:16.697053 20392 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38645
I20260620 19:21:16.697186 20392 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:16.697263 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20263
I20260620 19:21:16.697366 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:21:16.697377 20392 heartbeater.cc:511] Master 127.1.1.254:38645 requested a full tablet report, sending...
I20260620 19:21:16.697561 1031 external_mini_cluster.cc:1428] Setting key 63bcc4c7248702fa7fedb2f677b82f4f
I20260620 19:21:16.697893 20201 ts_manager.cc:194] Registered new tserver with Master: 4fff832381ed478aad0c631cf3a13337 (127.1.1.193:43939)
I20260620 19:21:16.698549 20201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.193@KRBTEST.COM'} at 127.1.1.193:59095
WARNING: no policy specified for kudu/127.1.1.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
Entry for principal kudu/127.1.1.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab.
I20260620 19:21:16.718068 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:16.788159 20397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:16.788304 20397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:16.788333 20397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:16.789299 20397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:16.789338 20397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:16.789404 20397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:16.790691 20397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.194.keytab
--principal=kudu/127.1.1.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:16.790858 20397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:16.791051 20397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:16.795374 20403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:16.795413 20397 server_base.cc:1034] running on GCE node
W20260620 19:21:16.795349 20402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.795351 20405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:16.795640 20397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:16.795873 20397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:16.797019 20397 hybrid_clock.cc:648] HybridClock initialized: now 1781983276797007 us; error 24 us; skew 500 ppm
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:16.799045 20397 init.cc:377] Logged in from keytab as kudu/127.1.1.194@KRBTEST.COM (short username kudu)
I20260620 19:21:16.799463 20397 webserver.cc:460] Webserver started at http://127.1.1.194:40653/ using document root <none> and password file <none>
I20260620 19:21:16.799678 20397 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:16.799742 20397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:16.799881 20397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:16.800755 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "138645e26eee4b279e0a97d025afe6d8"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "d004d140df253b821d9d2755b652fc06"
server_key_iv: "e9bacd9357f9309707a4bc2d2da3853a"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.801057 20397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "138645e26eee4b279e0a97d025afe6d8"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "d004d140df253b821d9d2755b652fc06"
server_key_iv: "e9bacd9357f9309707a4bc2d2da3853a"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.802217 20397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:21:16.802883 20412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.803057 20397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:16.803160 20397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "138645e26eee4b279e0a97d025afe6d8"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "d004d140df253b821d9d2755b652fc06"
server_key_iv: "e9bacd9357f9309707a4bc2d2da3853a"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.803243 20397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:16.815155 20397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:16.816619 20397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:16.816756 20397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:16.816942 20397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:16.817273 20397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:16.817332 20397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.817371 20397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:16.817399 20397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.822674 20397 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:43903
I20260620 19:21:16.822715 20525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:43903 every 8 connection(s)
I20260620 19:21:16.822978 20397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.194@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:16.831787 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20397
I20260620 19:21:16.831887 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:16.832098 1031 external_mini_cluster.cc:1428] Setting key fa2efb6af50f11a837b70d7f9c78d62c
I20260620 19:21:16.834265 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.823994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:34841 (local address 127.1.1.254:38645)
0620 19:21:16.824036 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.824038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.824049 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:21:16.824496 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.824497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.824546 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.824619 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:16.826793 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.827593 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.829045 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.829266 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.829331 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.831417 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:16.831418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:16.831418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:16.831426 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:16.832898 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.833365 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.833366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.833366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.833428 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.833683 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.833684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.833684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.833802 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:16.833867 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.834098 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.834187 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:16.834405 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.823560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38645 (local address 127.1.1.194:34841)
0620 19:21:16.823783 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.823818 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.824369 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.824656 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.824659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.826539 (+ 1880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:16.826698 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.826702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.827689 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.827691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.828950 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.828953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.829348 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.829349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.829460 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.829886 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:16.829895 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:16.831318 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:16.832994 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.833001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.833003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.833280 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.833494 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.833495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.833502 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.833609 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.833859 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:16.833864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:16.833980 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.834069 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.834214 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260620 19:21:16.834769 20526 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38645
I20260620 19:21:16.834846 20526 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:16.835011 20526 heartbeater.cc:511] Master 127.1.1.254:38645 requested a full tablet report, sending...
I20260620 19:21:16.835299 20201 ts_manager.cc:194] Registered new tserver with Master: 138645e26eee4b279e0a97d025afe6d8 (127.1.1.194:43903)
I20260620 19:21:16.835602 20201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.194@KRBTEST.COM'} at 127.1.1.194:34841
WARNING: no policy specified for kudu/127.1.1.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
Entry for principal kudu/127.1.1.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab.
I20260620 19:21:16.852919 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:16.928877 20531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:16.929042 20531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:16.929064 20531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:16.930159 20531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:16.930197 20531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:16.930255 20531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:16.931634 20531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.195.keytab
--principal=kudu/127.1.1.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:16.931866 20531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:16.932102 20531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:16.936688 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.936697 20537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:16.936745 20539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:16.936971 20531 server_base.cc:1034] running on GCE node
I20260620 19:21:16.937191 20531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:16.937443 20531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:16.938618 20531 hybrid_clock.cc:648] HybridClock initialized: now 1781983276938596 us; error 36 us; skew 500 ppm
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:16.940883 20531 init.cc:377] Logged in from keytab as kudu/127.1.1.195@KRBTEST.COM (short username kudu)
I20260620 19:21:16.941356 20531 webserver.cc:460] Webserver started at http://127.1.1.195:43031/ using document root <none> and password file <none>
I20260620 19:21:16.941565 20531 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:16.941627 20531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:16.941752 20531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:16.942694 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "6e2b1e3909944c16bacd0deb94d3aec2"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "afc314a870d8428bfdfe481536370096"
server_key_iv: "aac6f98ed68d1f60086d06d1c5c43eb6"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.943048 20531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "6e2b1e3909944c16bacd0deb94d3aec2"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "afc314a870d8428bfdfe481536370096"
server_key_iv: "aac6f98ed68d1f60086d06d1c5c43eb6"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.944267 20531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:16.944945 20546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.945122 20531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:16.945220 20531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "6e2b1e3909944c16bacd0deb94d3aec2"
format_stamp: "Formatted at 2026-06-20 19:21:16 on dist-test-slave-mwkb"
server_key: "afc314a870d8428bfdfe481536370096"
server_key_iv: "aac6f98ed68d1f60086d06d1c5c43eb6"
server_key_version: "encryptionkey@0"
I20260620 19:21:16.945303 20531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:16.958094 20531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:16.959595 20531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:16.959743 20531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:16.959949 20531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:16.960222 20531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:16.960275 20531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.960314 20531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:16.960341 20531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:16.965772 20531 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:34181
I20260620 19:21:16.965817 20659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:34181 every 8 connection(s)
I20260620 19:21:16.966081 20531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:21:16.967796 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20531
I20260620 19:21:16.967900 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:16.968082 1031 external_mini_cluster.cc:1428] Setting key 85e93e825af268a1d7d4623f1c1d2abc
Jun 20 19:21:16 dist-test-slave-mwkb krb5kdc[20156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983276, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.195@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:16.974107 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.966713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:42519 (local address 127.1.1.254:38645)
0620 19:21:16.966752 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.966754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.966936 (+ 182us) server_negotiation.cc:407] Connection header received
0620 19:21:16.967358 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.967359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.967371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.967413 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:16.968506 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.969112 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.969945 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.970126 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.970174 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.971889 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:16.971890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:16.971890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:16.971896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:16.972975 (+ 1079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.973381 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.973382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.973382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.973430 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.973642 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.973642 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.973643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.973736 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:16.973796 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.973996 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.974050 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:16.974268 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.966672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:38645 (local address 127.1.1.195:42519)
0620 19:21:16.966835 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.966867 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.967261 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.967479 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.967482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.968170 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:16.968419 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.968424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.969175 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.969176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.969880 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.969882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.970195 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.970195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.970282 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.970641 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:16.970648 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:16.971816 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:16.973042 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.973047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.973049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.973318 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.973490 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:16.973490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:16.973491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:16.973587 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:16.973801 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:16.973804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:16.973897 (+ 93us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.973969 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.974070 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260620 19:21:16.974759 20660 heartbeater.cc:347] Connected to a master server at 127.1.1.254:38645
I20260620 19:21:16.974828 20660 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:16.974941 20660 heartbeater.cc:511] Master 127.1.1.254:38645 requested a full tablet report, sending...
I20260620 19:21:16.975247 20201 ts_manager.cc:194] Registered new tserver with Master: 6e2b1e3909944c16bacd0deb94d3aec2 (127.1.1.195:34181)
I20260620 19:21:16.975549 20201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.1.195@KRBTEST.COM'} at 127.1.1.195:42519
I20260620 19:21:16.979687 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:16.985998 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.980935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.1.1.254:38645)
0620 19:21:16.980980 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.980982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.981049 (+ 67us) server_negotiation.cc:407] Connection header received
0620 19:21:16.981133 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.981134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.981142 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.981190 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:16.981638 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.982024 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.982831 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.982999 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.983051 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.984101 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:16.984102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:16.984102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:16.984106 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:16.985112 (+ 1006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.985415 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.985416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.985416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.985460 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:16.985612 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:16.985612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:16.985612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:16.985723 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:16.985791 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.985888 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.985938 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:21:16.987419 20201 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:16.988817 20201 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:16.994346 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34181 (local address 127.0.0.1:50732)
0620 19:21:16.990743 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.990770 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.990831 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.991105 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.991108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.991110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:16.991168 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.991169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992119 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.992119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.993316 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.993326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994090 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.994161 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994171 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.994231 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.994277 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260620 19:21:16.994453 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.1.1.194:43903)
0620 19:21:16.991137 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.991141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.991148 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:16.991178 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.991179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.991193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.991289 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:16.991365 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992391 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.993486 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994140 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994206 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994250 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.994325 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.994357 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260620 19:21:16.994562 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.1.1.195:34181)
0620 19:21:16.990958 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.990962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.990970 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:16.991005 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.991007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.991018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.991082 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:16.991238 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992026 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.993413 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994004 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994086 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994132 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.994468 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.994503 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260620 19:21:16.994850 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:43939 (local address 127.0.0.1:49018)
0620 19:21:16.991131 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.991139 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.991190 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.991400 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.991401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.991402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:16.991444 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.991446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992598 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.992599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.993692 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.993695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994647 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.994688 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994691 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.994730 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.994788 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260620 19:21:16.994911 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.1.1.193:43939)
0620 19:21:16.990999 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:16.991003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:16.991189 (+ 186us) server_negotiation.cc:407] Connection header received
0620 19:21:16.991253 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:16.991255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:16.991267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:16.991326 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:16.991545 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992485 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.993780 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994394 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994558 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:16.994768 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:21:16.994807 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260620 19:21:16.995172 20594 tablet_service.cc:1467] Processing CreateTablet for tablet aa75bd9fdd2148eba314a03d910823bd (DEFAULT_TABLE table=test-table [id=659c6107beed4a9c816bdce8d86cd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:16.995275 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:16.990604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:43903 (local address 127.0.0.1:47356)
0620 19:21:16.990742 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:16.990769 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:16.990827 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:16.991286 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:16.991287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:16.991289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:16.991315 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.991316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.992495 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.992497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.993399 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:16.993402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:16.994216 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:16.994217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:16.994263 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:16.994266 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:16.994305 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:21:16.994339 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260620 19:21:16.995496 20594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa75bd9fdd2148eba314a03d910823bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:16.995556 20326 tablet_service.cc:1467] Processing CreateTablet for tablet aa75bd9fdd2148eba314a03d910823bd (DEFAULT_TABLE table=test-table [id=659c6107beed4a9c816bdce8d86cd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:16.995870 20326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa75bd9fdd2148eba314a03d910823bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:16.997643 20460 tablet_service.cc:1467] Processing CreateTablet for tablet aa75bd9fdd2148eba314a03d910823bd (DEFAULT_TABLE table=test-table [id=659c6107beed4a9c816bdce8d86cd21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:16.997941 20460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa75bd9fdd2148eba314a03d910823bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:16.998050 20674 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Bootstrap starting.
I20260620 19:21:16.998642 20674 tablet_bootstrap.cc:654] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:16.999161 20674 log.cc:824] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:16.999910 20674 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: No bootstrap required, opened a new log
I20260620 19:21:16.999977 20674 ts_tablet_manager.cc:1397] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:17.001122 20677 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Bootstrap starting.
I20260620 19:21:17.001410 20675 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: Bootstrap starting.
I20260620 19:21:17.001844 20677 tablet_bootstrap.cc:654] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.001991 20675 tablet_bootstrap.cc:654] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.001899 20674 raft_consensus.cc:348] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.002048 20674 raft_consensus.cc:374] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.002089 20674 raft_consensus.cc:729] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2b1e3909944c16bacd0deb94d3aec2, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.002216 20674 consensus_queue.cc:260] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.002357 20677 log.cc:824] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.002475 20674 ts_tablet_manager.cc:1428] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:21:17.002499 20675 log.cc:824] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.002606 20660 heartbeater.cc:503] Master 127.1.1.254:38645 was elected leader, sending a full tablet report...
I20260620 19:21:17.003080 20677 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: No bootstrap required, opened a new log
I20260620 19:21:17.003147 20677 ts_tablet_manager.cc:1397] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:17.003409 20675 tablet_bootstrap.cc:492] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: No bootstrap required, opened a new log
I20260620 19:21:17.003496 20675 ts_tablet_manager.cc:1397] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:17.004522 20677 raft_consensus.cc:348] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.004655 20677 raft_consensus.cc:374] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.004693 20677 raft_consensus.cc:729] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138645e26eee4b279e0a97d025afe6d8, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.004796 20677 consensus_queue.cc:260] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.004825 20675 raft_consensus.cc:348] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.004978 20675 raft_consensus.cc:374] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.005021 20675 raft_consensus.cc:729] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff832381ed478aad0c631cf3a13337, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.005051 20526 heartbeater.cc:503] Master 127.1.1.254:38645 was elected leader, sending a full tablet report...
I20260620 19:21:17.005036 20677 ts_tablet_manager.cc:1428] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:17.005125 20675 consensus_queue.cc:260] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.005366 20675 ts_tablet_manager.cc:1428] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:17.005420 20392 heartbeater.cc:503] Master 127.1.1.254:38645 was elected leader, sending a full tablet report...
W20260620 19:21:17.073448 20527 tablet.cc:2369] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:17.078619 20682 raft_consensus.cc:482] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:17.078706 20682 raft_consensus.cc:504] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.079073 20682 leader_election.cc:288] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e2b1e3909944c16bacd0deb94d3aec2 (127.1.1.195:34181), 138645e26eee4b279e0a97d025afe6d8 (127.1.1.194:43903)
I20260620 19:21:17.081931 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.079325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:35817 (local address 127.1.1.194:43903)
0620 19:21:17.079366 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.079370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.079382 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:21:17.079407 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.079408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.079417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.079466 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.079632 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.080256 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081243 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.081600 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081677 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.081686 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.081841 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.081875 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:21:17.081931 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.079248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:43903 (local address 127.1.1.193:35817)
0620 19:21:17.079295 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.079307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.079351 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.079490 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.079491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.079492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.079563 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.079566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.080340 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.080341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.081162 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.081164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.081686 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.081761 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.081764 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.081835 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.081871 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 19:21:17.082197 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.079248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:34181 (local address 127.1.1.193:50415)
0620 19:21:17.079621 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.079632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.079688 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.079838 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.079839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.079840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.079903 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.079904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.080510 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.080511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.081433 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.081436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.081962 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.082031 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.082035 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.082085 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.082119 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":154,"thread_start_us":109,"threads_started":1}
I20260620 19:21:17.082254 20671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.079303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:50415 (local address 127.1.1.195:34181)
0620 19:21:17.079375 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.079377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.079725 (+ 348us) server_negotiation.cc:407] Connection header received
0620 19:21:17.079752 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.079753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.079766 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.079816 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.079968 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.080398 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081532 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.081881 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.081958 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.081966 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.082157 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.082194 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:21:17.082278 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa75bd9fdd2148eba314a03d910823bd" candidate_uuid: "4fff832381ed478aad0c631cf3a13337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138645e26eee4b279e0a97d025afe6d8" is_pre_election: true
I20260620 19:21:17.082418 20480 raft_consensus.cc:2454] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff832381ed478aad0c631cf3a13337 in term 0.
I20260620 19:21:17.082506 20614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa75bd9fdd2148eba314a03d910823bd" candidate_uuid: "4fff832381ed478aad0c631cf3a13337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" is_pre_election: true
I20260620 19:21:17.082677 20614 raft_consensus.cc:2454] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff832381ed478aad0c631cf3a13337 in term 0.
I20260620 19:21:17.082660 20281 leader_election.cc:302] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138645e26eee4b279e0a97d025afe6d8, 4fff832381ed478aad0c631cf3a13337; no voters:
I20260620 19:21:17.082814 20682 raft_consensus.cc:2781] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:17.082880 20682 raft_consensus.cc:482] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:17.082903 20682 raft_consensus.cc:3037] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.083540 20682 raft_consensus.cc:504] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.083670 20682 leader_election.cc:288] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [CANDIDATE]: Term 1 election: Requested vote from peers 6e2b1e3909944c16bacd0deb94d3aec2 (127.1.1.195:34181), 138645e26eee4b279e0a97d025afe6d8 (127.1.1.194:43903)
I20260620 19:21:17.083848 20614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa75bd9fdd2148eba314a03d910823bd" candidate_uuid: "4fff832381ed478aad0c631cf3a13337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e2b1e3909944c16bacd0deb94d3aec2"
I20260620 19:21:17.083869 20480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa75bd9fdd2148eba314a03d910823bd" candidate_uuid: "4fff832381ed478aad0c631cf3a13337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138645e26eee4b279e0a97d025afe6d8"
I20260620 19:21:17.083920 20614 raft_consensus.cc:3037] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.083941 20480 raft_consensus.cc:3037] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.084571 20614 raft_consensus.cc:2454] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff832381ed478aad0c631cf3a13337 in term 1.
I20260620 19:21:17.084673 20480 raft_consensus.cc:2454] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff832381ed478aad0c631cf3a13337 in term 1.
I20260620 19:21:17.084798 20282 leader_election.cc:302] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff832381ed478aad0c631cf3a13337, 6e2b1e3909944c16bacd0deb94d3aec2; no voters:
I20260620 19:21:17.084882 20682 raft_consensus.cc:2781] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:17.085021 20682 raft_consensus.cc:686] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [term 1 LEADER]: Becoming Leader. State: Replica: 4fff832381ed478aad0c631cf3a13337, State: Running, Role: LEADER
I20260620 19:21:17.085150 20682 consensus_queue.cc:237] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } }
I20260620 19:21:17.085843 20200 catalog_manager.cc:5495] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fff832381ed478aad0c631cf3a13337 (127.1.1.193). New cstate: current_term: 1 leader_uuid: "4fff832381ed478aad0c631cf3a13337" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fff832381ed478aad0c631cf3a13337" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 43939 } health_report { overall_health: HEALTHY } } }
I20260620 19:21:17.110965 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.108708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.1.1.193:43939)
0620 19:21:17.108767 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.108769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.108774 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:21:17.108797 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.108797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.108811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.108874 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:17.108975 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.109477 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.110247 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.110431 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.110503 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.110678 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:17.110766 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:17.110855 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.110878 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.110896 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:21:17.112605 20480 raft_consensus.cc:1261] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 1 FOLLOWER]: Refusing update from remote peer 4fff832381ed478aad0c631cf3a13337: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:17.112617 20614 raft_consensus.cc:1261] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 1 FOLLOWER]: Refusing update from remote peer 4fff832381ed478aad0c631cf3a13337: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:17.112982 20682 consensus_queue.cc:1035] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "138645e26eee4b279e0a97d025afe6d8" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 43903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:17.113144 20684 consensus_queue.cc:1035] T aa75bd9fdd2148eba314a03d910823bd P 4fff832381ed478aad0c631cf3a13337 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e2b1e3909944c16bacd0deb94d3aec2" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 34181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:17.115914 20688 mvcc.cc:205] Tried to move back new op lower bound from 7299003503050530816 to 7299003502941249536. Current Snapshot: MvccSnapshot[applied={T|T < 7299003503050530816 or (T in {7299003503050530816})}]
I20260620 19:21:17.116163 20689 mvcc.cc:205] Tried to move back new op lower bound from 7299003503050530816 to 7299003502941249536. Current Snapshot: MvccSnapshot[applied={T|T < 7299003503050530816}]
I20260620 19:21:17.117372 20200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:17.117455 20200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:17.118170 20200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c250adb92d0e40ec973346c62c1cfa7c: Sending DeleteTablet for 3 replicas of tablet aa75bd9fdd2148eba314a03d910823bd
I20260620 19:21:17.118491 20594 tablet_service.cc:1514] Processing DeleteTablet for tablet aa75bd9fdd2148eba314a03d910823bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:50732
I20260620 19:21:17.118530 20460 tablet_service.cc:1514] Processing DeleteTablet for tablet aa75bd9fdd2148eba314a03d910823bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:47356
I20260620 19:21:17.118700 20696 tablet_replica.cc:330] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: stopping tablet replica
I20260620 19:21:17.118790 20696 raft_consensus.cc:2229] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:17.118818 20697 tablet_replica.cc:330] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: stopping tablet replica
I20260620 19:21:17.118850 20326 tablet_service.cc:1514] Processing DeleteTablet for tablet aa75bd9fdd2148eba314a03d910823bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='kudu', principal='kudu/127.1.1.254@KRBTEST.COM'} at 127.0.0.1:49018
I20260620 19:21:17.118875 20696 raft_consensus.cc:2258] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:17.118885 20697 raft_consensus.cc:2229] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:17.118956 20697 raft_consensus.cc:2258] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:17.119174 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20263
I20260620 19:21:17.120088 20696 ts_tablet_manager.cc:1905] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:17.120088 20697 ts_tablet_manager.cc:1905] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:17.121676 20697 ts_tablet_manager.cc:1918] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:17.121759 20697 log.cc:1196] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/aa75bd9fdd2148eba314a03d910823bd
I20260620 19:21:17.121824 20696 ts_tablet_manager.cc:1918] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:17.121901 20696 log.cc:1196] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/aa75bd9fdd2148eba314a03d910823bd
I20260620 19:21:17.121995 20697 ts_tablet_manager.cc:1939] T aa75bd9fdd2148eba314a03d910823bd P 138645e26eee4b279e0a97d025afe6d8: Deleting consensus metadata
I20260620 19:21:17.122207 20696 ts_tablet_manager.cc:1939] T aa75bd9fdd2148eba314a03d910823bd P 6e2b1e3909944c16bacd0deb94d3aec2: Deleting consensus metadata
I20260620 19:21:17.122430 20187 catalog_manager.cc:4841] TS 138645e26eee4b279e0a97d025afe6d8 (127.1.1.194:43903): tablet aa75bd9fdd2148eba314a03d910823bd (table test-table [id=659c6107beed4a9c816bdce8d86cd21a]) successfully deleted
I20260620 19:21:17.122558 20188 catalog_manager.cc:4841] TS 6e2b1e3909944c16bacd0deb94d3aec2 (127.1.1.195:34181): tablet aa75bd9fdd2148eba314a03d910823bd (table test-table [id=659c6107beed4a9c816bdce8d86cd21a]) successfully deleted
W20260620 19:21:17.124948 20186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:43939 (error 108)
I20260620 19:21:17.125396 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.125291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:43939 (local address 127.0.0.1:49028)
0620 19:21:17.125343 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.125373 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:21:17.125532 20186 catalog_manager.cc:4584] TS 4fff832381ed478aad0c631cf3a13337 (127.1.1.193:43939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa75bd9fdd2148eba314a03d910823bd: Network error: Client connection negotiation failed: client connection to 127.1.1.193:43939: connect: Connection refused (error 111)
I20260620 19:21:17.125558 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20397
I20260620 19:21:17.130460 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20531
I20260620 19:21:17.135224 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20169
2026-06-20T19:21:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1758 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:17Z Disabled control of system clock
I20260620 19:21:17.160054 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:42093
--webserver_interface=127.1.1.254
--webserver_port=0
--builtin_ntp_servers=127.1.1.212:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:17.231940 20703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:17.232127 20703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:17.232147 20703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:17.233263 20703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:17.233299 20703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:17.233314 20703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:17.233326 20703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:17.233338 20703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:17.234570 20703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:42093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:17.234732 20703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:17.234946 20703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:17.239209 20708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.239224 20711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.239260 20703 server_base.cc:1034] running on GCE node
W20260620 19:21:17.239224 20709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.239694 20703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:17.239931 20703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:17.241108 20703 hybrid_clock.cc:648] HybridClock initialized: now 1781983277241047 us; error 35 us; skew 500 ppm
I20260620 19:21:17.242094 20703 webserver.cc:460] Webserver started at http://127.1.1.254:40729/ using document root <none> and password file <none>
I20260620 19:21:17.242309 20703 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:17.242354 20703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:17.242479 20703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:17.243434 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "22772ca92b8b4dfe90446367baf03a3c"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f4e63c4d1ca996c881cac6d71209236a"
server_key_iv: "3dc5bc36e53ae6d7c45796704eb41f20"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.243773 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "22772ca92b8b4dfe90446367baf03a3c"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f4e63c4d1ca996c881cac6d71209236a"
server_key_iv: "3dc5bc36e53ae6d7c45796704eb41f20"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.244964 20703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:17.245687 20717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.245854 20703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.245934 20703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "22772ca92b8b4dfe90446367baf03a3c"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f4e63c4d1ca996c881cac6d71209236a"
server_key_iv: "3dc5bc36e53ae6d7c45796704eb41f20"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.246018 20703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:17.271411 20703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:17.271656 20703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:17.271780 20703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:17.275544 20703 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:42093
I20260620 19:21:17.275596 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:42093 every 8 connection(s)
I20260620 19:21:17.275872 20703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:17.276674 20770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:17.278541 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: Bootstrap starting.
I20260620 19:21:17.278977 20770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.279196 20770 log.cc:824] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.279783 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: No bootstrap required, opened a new log
I20260620 19:21:17.281193 20770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } }
I20260620 19:21:17.281297 20770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.281316 20770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22772ca92b8b4dfe90446367baf03a3c, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.281397 20770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } }
I20260620 19:21:17.281438 20770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:17.281455 20770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:17.281483 20770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.281915 20770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } }
I20260620 19:21:17.282021 20770 leader_election.cc:302] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22772ca92b8b4dfe90446367baf03a3c; no voters:
I20260620 19:21:17.282169 20770 leader_election.cc:288] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:17.282231 20773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:17.282392 20773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [term 1 LEADER]: Becoming Leader. State: Replica: 22772ca92b8b4dfe90446367baf03a3c, State: Running, Role: LEADER
I20260620 19:21:17.282423 20770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:17.282478 20773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } }
I20260620 19:21:17.283116 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22772ca92b8b4dfe90446367baf03a3c. Latest consensus state: current_term: 1 leader_uuid: "22772ca92b8b4dfe90446367baf03a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } } }
I20260620 19:21:17.283217 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:17.283337 20774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22772ca92b8b4dfe90446367baf03a3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22772ca92b8b4dfe90446367baf03a3c" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 42093 } } }
I20260620 19:21:17.283392 20774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:17.283496 20782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:17.283864 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20703
I20260620 19:21:17.283962 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:17.283962 20782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:17.284143 1031 external_mini_cluster.cc:1428] Setting key decc16673683bce2abe0ecfd38230940
I20260620 19:21:17.285427 20782 catalog_manager.cc:1347] Generated new cluster ID: c7ea3ed653f944a1a1a9080e9e6ece0a
I20260620 19:21:17.285475 20782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:17.289806 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.284469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.1.1.254:42093)
0620 19:21:17.285009 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.285016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.285041 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:21:17.285154 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.285160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.285220 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.285347 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:17.285633 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.286846 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.287883 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.288405 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.288495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.289057 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:17.289060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:17.289060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:17.289237 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:17.289273 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:17.289285 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:17.289337 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.289443 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.289508 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":62,"threads_started":1}
I20260620 19:21:17.298802 20782 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:17.299197 20782 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:17.303354 20782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: Generated new TSK 0
I20260620 19:21:17.303530 20782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:17.307327 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.193:0
--local_ip_for_outbound_sockets=127.1.1.193
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--builtin_ntp_servers=127.1.1.212:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:17.381853 20794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:17.382074 20794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:17.382150 20794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:17.383476 20794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:17.383534 20794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:17.383576 20794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.193
I20260620 19:21:17.384915 20794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.1.193
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.193
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:17.385157 20794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:17.385407 20794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:17.390177 20800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.390187 20802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.390244 20799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.390772 20794 server_base.cc:1034] running on GCE node
I20260620 19:21:17.390949 20794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:17.391170 20794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:17.392328 20794 hybrid_clock.cc:648] HybridClock initialized: now 1781983277392316 us; error 22 us; skew 500 ppm
I20260620 19:21:17.393604 20794 webserver.cc:460] Webserver started at http://127.1.1.193:44595/ using document root <none> and password file <none>
I20260620 19:21:17.393784 20794 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:17.393836 20794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:17.393916 20794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:17.394779 20794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data/instance:
uuid: "581f6b265d7b42c282cca03b716ae6a2"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "57202fc1938da2ff09941f23886a2202"
server_key_iv: "421bcc35305585b4eb02d0a68c981353"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.395078 20794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance:
uuid: "581f6b265d7b42c282cca03b716ae6a2"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "57202fc1938da2ff09941f23886a2202"
server_key_iv: "421bcc35305585b4eb02d0a68c981353"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.396181 20794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:17.396759 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.396912 20794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.396999 20794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal
uuid: "581f6b265d7b42c282cca03b716ae6a2"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "57202fc1938da2ff09941f23886a2202"
server_key_iv: "421bcc35305585b4eb02d0a68c981353"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.397099 20794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:17.420574 20794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:17.420830 20794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:17.420964 20794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:17.421214 20794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:17.421504 20794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:17.421552 20794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.421582 20794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:17.421604 20794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.426937 20794 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.193:34945
I20260620 19:21:17.426970 20921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.193:34945 every 8 connection(s)
I20260620 19:21:17.427253 20794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/data/info.pb
I20260620 19:21:17.431293 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.427857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.193:38371 (local address 127.1.1.254:42093)
0620 19:21:17.427933 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.427935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.428129 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:21:17.428528 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.428529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.428538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.428583 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:17.429110 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.429547 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.430325 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.430481 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.430529 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.431095 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:17.431096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:17.431096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:17.431099 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:17.431111 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:17.431114 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:17.431154 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.431207 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.431227 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:21:17.431468 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.427816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:42093 (local address 127.1.1.193:38371)
0620 19:21:17.427994 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.428038 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.428450 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.428625 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.428628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.428634 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:17.429024 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.429027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.429617 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.429617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.430258 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.430259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.430545 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.430546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.430631 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.430986 (+ 355us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:17.430986 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:17.431008 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:17.431026 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:17.431157 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:17.431158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:17.431160 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.431198 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.431256 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260620 19:21:17.431938 20922 heartbeater.cc:347] Connected to a master server at 127.1.1.254:42093
I20260620 19:21:17.432000 20922 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:17.432152 20922 heartbeater.cc:511] Master 127.1.1.254:42093 requested a full tablet report, sending...
I20260620 19:21:17.432574 20734 ts_manager.cc:194] Registered new tserver with Master: 581f6b265d7b42c282cca03b716ae6a2 (127.1.1.193:34945)
I20260620 19:21:17.433197 20734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.193:38371
I20260620 19:21:17.435619 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20794
I20260620 19:21:17.435730 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-0/wal/instance
I20260620 19:21:17.435878 1031 external_mini_cluster.cc:1428] Setting key 7d0a05ebb9a788d523be3509a2400828
I20260620 19:21:17.436576 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.194:0
--local_ip_for_outbound_sockets=127.1.1.194
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--builtin_ntp_servers=127.1.1.212:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:17.507567 20925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:17.507716 20925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:17.507735 20925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:17.508623 20925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:17.508651 20925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:17.508670 20925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.194
I20260620 19:21:17.509826 20925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.1.194
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.194
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:17.510043 20925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:17.510329 20925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:17.514804 20930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.514839 20933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.514832 20931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.514972 20925 server_base.cc:1034] running on GCE node
I20260620 19:21:17.515193 20925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:17.515431 20925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:17.516587 20925 hybrid_clock.cc:648] HybridClock initialized: now 1781983277516576 us; error 38 us; skew 500 ppm
I20260620 19:21:17.517733 20925 webserver.cc:460] Webserver started at http://127.1.1.194:37409/ using document root <none> and password file <none>
I20260620 19:21:17.517948 20925 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:17.518020 20925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:17.518163 20925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:17.519039 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data/instance:
uuid: "7831d89598944a66b3b4d71872d394c0"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f9d51c3085f6818bd6185b4969e057f9"
server_key_iv: "ab2b7ac6d8b8ab3d13001f805f21764d"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.519377 20925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance:
uuid: "7831d89598944a66b3b4d71872d394c0"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f9d51c3085f6818bd6185b4969e057f9"
server_key_iv: "ab2b7ac6d8b8ab3d13001f805f21764d"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.520588 20925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:17.521246 20939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.521402 20925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:17.521502 20925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal
uuid: "7831d89598944a66b3b4d71872d394c0"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "f9d51c3085f6818bd6185b4969e057f9"
server_key_iv: "ab2b7ac6d8b8ab3d13001f805f21764d"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.521574 20925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:17.534453 20925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:17.534693 20925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:17.534821 20925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:17.535002 20925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:17.535270 20925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:17.535318 20925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.535352 20925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:17.535377 20925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.540393 20925 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.194:35763
I20260620 19:21:17.540422 21052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.194:35763 every 8 connection(s)
I20260620 19:21:17.540695 20925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/data/info.pb
I20260620 19:21:17.544972 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.541340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:41481 (local address 127.1.1.254:42093)
0620 19:21:17.541416 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.541417 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.541579 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:21:17.541949 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.541950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.541959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.542002 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:17.542550 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.543039 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.543949 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.544121 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.544167 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.544789 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:17.544790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:17.544790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:17.544794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:17.544805 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:17.544809 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:17.544848 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.544900 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.544920 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:21:17.545142 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.541304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:42093 (local address 127.1.1.194:41481)
0620 19:21:17.541468 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.541493 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.541877 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.542051 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.542054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.542061 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:17.542488 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.542492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.543116 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.543116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.543879 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.543881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.544189 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.544190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.544282 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.544677 (+ 395us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:17.544677 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:17.544700 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:17.544718 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:17.544848 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:17.544850 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:17.544852 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.544893 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.544955 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260620 19:21:17.545614 21053 heartbeater.cc:347] Connected to a master server at 127.1.1.254:42093
I20260620 19:21:17.545704 21053 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:17.545820 21053 heartbeater.cc:511] Master 127.1.1.254:42093 requested a full tablet report, sending...
I20260620 19:21:17.546135 20734 ts_manager.cc:194] Registered new tserver with Master: 7831d89598944a66b3b4d71872d394c0 (127.1.1.194:35763)
I20260620 19:21:17.546449 20734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.194:41481
I20260620 19:21:17.550554 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 20925
I20260620 19:21:17.550647 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal/instance
I20260620 19:21:17.550812 1031 external_mini_cluster.cc:1428] Setting key d3ff361aafdcaba1fc32716343ca7dd3
I20260620 19:21:17.551548 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.1.195:0
--local_ip_for_outbound_sockets=127.1.1.195
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--builtin_ntp_servers=127.1.1.212:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:21:17.624017 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:17.624194 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:17.624222 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:17.625311 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:17.625365 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:17.625401 21056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.1.195
I20260620 19:21:17.626637 21056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.1.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.1.195
--webserver_port=0
--tserver_master_addrs=127.1.1.254:42093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.1.195
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:17.626850 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:17.627089 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:17.631471 21064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.631560 21056 server_base.cc:1034] running on GCE node
W20260620 19:21:17.631485 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:17.631480 21062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:17.631878 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:17.632114 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:17.633271 21056 hybrid_clock.cc:648] HybridClock initialized: now 1781983277633267 us; error 30 us; skew 500 ppm
I20260620 19:21:17.634346 21056 webserver.cc:460] Webserver started at http://127.1.1.195:43825/ using document root <none> and password file <none>
I20260620 19:21:17.634549 21056 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:17.634608 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:17.634719 21056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:17.635574 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data/instance:
uuid: "a4e9f491cf8c4fd9aa59115eac440037"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "6b1d6abe40c799116a07d321c2854338"
server_key_iv: "ade37b2cafcc7dad3360a0bcfe054be3"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.635891 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance:
uuid: "a4e9f491cf8c4fd9aa59115eac440037"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "6b1d6abe40c799116a07d321c2854338"
server_key_iv: "ade37b2cafcc7dad3360a0bcfe054be3"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.637033 21056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:17.637718 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.637884 21056 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:17.637964 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal
uuid: "a4e9f491cf8c4fd9aa59115eac440037"
format_stamp: "Formatted at 2026-06-20 19:21:17 on dist-test-slave-mwkb"
server_key: "6b1d6abe40c799116a07d321c2854338"
server_key_iv: "ade37b2cafcc7dad3360a0bcfe054be3"
server_key_version: "encryptionkey@0"
I20260620 19:21:17.638054 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:17.653730 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:17.654000 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:17.654134 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:17.654314 21056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:21:17.654584 21056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:21:17.654630 21056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.654660 21056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:21:17.654682 21056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:17.659668 21056 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.195:33135
I20260620 19:21:17.659734 21183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.195:33135 every 8 connection(s)
I20260620 19:21:17.660001 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/data/info.pb
I20260620 19:21:17.664155 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.660612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.195:35955 (local address 127.1.1.254:42093)
0620 19:21:17.660688 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.660691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.660864 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:21:17.661261 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.661262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.661272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.661330 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:17.661813 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.662220 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.663041 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.663212 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.663260 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.663991 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:17.663991 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:17.663992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:17.663995 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:17.664003 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:17.664006 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:17.664037 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.664088 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.664107 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:21:17.664325 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.660571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:42093 (local address 127.1.1.195:35955)
0620 19:21:17.660729 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.660766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.661157 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.661377 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.661380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.661386 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:17.661759 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.661761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.662287 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.662287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.662972 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.662974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.663282 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.663283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.663380 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.663894 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:21:17.663895 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:17.663916 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:21:17.663933 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:17.664039 (+ 106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:17.664041 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:17.664043 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.664081 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.664139 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 19:21:17.664716 21184 heartbeater.cc:347] Connected to a master server at 127.1.1.254:42093
I20260620 19:21:17.664803 21184 heartbeater.cc:464] Registering TS with master...
I20260620 19:21:17.664948 21184 heartbeater.cc:511] Master 127.1.1.254:42093 requested a full tablet report, sending...
I20260620 19:21:17.665307 20734 ts_manager.cc:194] Registered new tserver with Master: a4e9f491cf8c4fd9aa59115eac440037 (127.1.1.195:33135)
I20260620 19:21:17.665628 20734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.1.195:35955
I20260620 19:21:17.665951 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21056
I20260620 19:21:17.666049 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal/instance
I20260620 19:21:17.666221 1031 external_mini_cluster.cc:1428] Setting key 413740946aedb33b402df90be8af6912
I20260620 19:21:17.666934 1031 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:21:17.670167 20792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.668046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.1.1.254:42093)
0620 19:21:17.668112 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.668114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.668157 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:21:17.668196 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.668197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.668204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.668243 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:17.668355 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.668715 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.669391 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.669533 (+ 142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.669578 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.669992 (+ 414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:17.669993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:17.669993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:17.669996 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:17.670005 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:17.670008 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:17.670047 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.670076 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.670091 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:21:17.671417 20734 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:21:17.672708 20734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:21:17.678040 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:33135 (local address 127.0.0.1:39950)
0620 19:21:17.674796 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.674839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.674885 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.675133 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.675134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.675136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.675175 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.675176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.676290 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.676290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.677057 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.677059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.677831 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677831 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.677894 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.677906 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.677959 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.678001 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":71,"threads_started":1}
I20260620 19:21:17.678251 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39950 (local address 127.1.1.195:33135)
0620 19:21:17.674985 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.674988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.674994 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:21:17.675023 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.675025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.675033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.675110 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.675247 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.676218 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677155 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.677753 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677817 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.677880 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.678144 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.678182 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":76,"threads_started":1}
I20260620 19:21:17.678838 21192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.194:35763 (local address 127.0.0.1:54180)
0620 19:21:17.674804 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.674828 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.674885 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.675099 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.675101 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.675103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.675154 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.675156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.677207 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.677876 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.677877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.678629 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.678630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.678682 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.678685 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.678747 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.678783 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":61,"threads_started":1}
I20260620 19:21:17.678750 21118 tablet_service.cc:1467] Processing CreateTablet for tablet 761820c53460443aaf8874c626da6403 (DEFAULT_TABLE table=test-table [id=4ddd5e2e450643deb4c00ee9b6ffb5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:17.678864 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.1.1.194:35763)
0620 19:21:17.674839 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.674843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.674880 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:21:17.674990 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.674992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.675005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.675077 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.675221 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.676225 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677939 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.678559 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.678613 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.678682 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.678759 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.678793 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260620 19:21:17.679032 21118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761820c53460443aaf8874c626da6403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:17.679486 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34945 (local address 127.0.0.1:47278)
0620 19:21:17.675172 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.675181 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.675225 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.676454 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.676455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.676457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.676486 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.676487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.677285 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.677286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.678482 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.678485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.679261 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.679262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.679338 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.679341 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.679394 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.679429 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":56,"threads_started":1}
I20260620 19:21:17.679524 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.674785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.1.1.193:34945)
0620 19:21:17.674938 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.674941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.676315 (+ 1374us) server_negotiation.cc:407] Connection header received
0620 19:21:17.676359 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.676361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.676382 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.676435 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.676549 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.677205 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.678571 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.679183 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.679260 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.679359 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.679416 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.679448 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260620 19:21:17.679459 20987 tablet_service.cc:1467] Processing CreateTablet for tablet 761820c53460443aaf8874c626da6403 (DEFAULT_TABLE table=test-table [id=4ddd5e2e450643deb4c00ee9b6ffb5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:17.679744 20987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761820c53460443aaf8874c626da6403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:17.681855 21198 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Bootstrap starting.
I20260620 19:21:17.682333 21198 tablet_bootstrap.cc:654] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.682513 20856 tablet_service.cc:1467] Processing CreateTablet for tablet 761820c53460443aaf8874c626da6403 (DEFAULT_TABLE table=test-table [id=4ddd5e2e450643deb4c00ee9b6ffb5fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:21:17.682667 21198 log.cc:824] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.682745 21199 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Bootstrap starting.
I20260620 19:21:17.682857 20856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761820c53460443aaf8874c626da6403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:17.683290 21199 tablet_bootstrap.cc:654] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.683570 21199 log.cc:824] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.684307 21199 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: No bootstrap required, opened a new log
I20260620 19:21:17.684346 21199 ts_tablet_manager.cc:1397] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:17.685361 21202 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: Bootstrap starting.
I20260620 19:21:17.685899 21202 tablet_bootstrap.cc:654] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:17.686164 21202 log.cc:824] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:17.686220 21198 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: No bootstrap required, opened a new log
I20260620 19:21:17.686281 21198 ts_tablet_manager.cc:1397] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:21:17.686365 21199 raft_consensus.cc:348] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.686506 21199 raft_consensus.cc:374] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.686537 21199 raft_consensus.cc:729] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7831d89598944a66b3b4d71872d394c0, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.686659 21199 consensus_queue.cc:260] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.686815 21202 tablet_bootstrap.cc:492] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: No bootstrap required, opened a new log
I20260620 19:21:17.686887 21202 ts_tablet_manager.cc:1397] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:21:17.686929 21199 ts_tablet_manager.cc:1428] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:21:17.687062 21053 heartbeater.cc:503] Master 127.1.1.254:42093 was elected leader, sending a full tablet report...
I20260620 19:21:17.688280 21198 raft_consensus.cc:348] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.688401 21198 raft_consensus.cc:374] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.688437 21198 raft_consensus.cc:729] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e9f491cf8c4fd9aa59115eac440037, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.688536 21198 consensus_queue.cc:260] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.688647 21202 raft_consensus.cc:348] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.688755 21198 ts_tablet_manager.cc:1428] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:21:17.688766 21202 raft_consensus.cc:374] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:17.688766 21184 heartbeater.cc:503] Master 127.1.1.254:42093 was elected leader, sending a full tablet report...
I20260620 19:21:17.688803 21202 raft_consensus.cc:729] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581f6b265d7b42c282cca03b716ae6a2, State: Initialized, Role: FOLLOWER
I20260620 19:21:17.688907 21202 consensus_queue.cc:260] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.689144 21202 ts_tablet_manager.cc:1428] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:21:17.689181 20922 heartbeater.cc:503] Master 127.1.1.254:42093 was elected leader, sending a full tablet report...
I20260620 19:21:17.755131 21204 raft_consensus.cc:482] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:21:17.755254 21204 raft_consensus.cc:504] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.755648 21204 leader_election.cc:288] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e9f491cf8c4fd9aa59115eac440037 (127.1.1.195:33135), 581f6b265d7b42c282cca03b716ae6a2 (127.1.1.193:34945)
I20260620 19:21:17.758498 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.756002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:49767 (local address 127.1.1.193:34945)
0620 19:21:17.756059 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.756061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.756152 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:21:17.756205 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.756206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.756218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.756273 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.756461 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.756915 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.757808 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.758158 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.758210 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.758218 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.758386 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.758427 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:17.758510 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.755939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34945 (local address 127.1.1.194:49767)
0620 19:21:17.756089 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.756098 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.756146 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.756295 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.756295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.756297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.756377 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.756378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.756992 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.756993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.757726 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.757727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.758238 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.758240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.758315 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.758319 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.758366 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.758420 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260620 19:21:17.758534 21194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.755913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.1.194:52011 (local address 127.1.1.195:33135)
0620 19:21:17.755957 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.755959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.755968 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:17.756035 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.756036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.756046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.756101 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:21:17.756296 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.756814 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.757852 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.758175 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.758260 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.758268 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.758433 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.758472 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:21:17.758527 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.755829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.195:33135 (local address 127.1.1.194:52011)
0620 19:21:17.755896 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.755910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:17.755965 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:17.756121 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:17.756122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:17.756123 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:21:17.756207 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.756209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.756888 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.756889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.757729 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:17.757731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.758284 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.758284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:17.758341 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.758344 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:21:17.758414 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:21:17.758450 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":14}
I20260620 19:21:17.758927 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761820c53460443aaf8874c626da6403" candidate_uuid: "7831d89598944a66b3b4d71872d394c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581f6b265d7b42c282cca03b716ae6a2" is_pre_election: true
I20260620 19:21:17.759099 20876 raft_consensus.cc:2454] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7831d89598944a66b3b4d71872d394c0 in term 0.
I20260620 19:21:17.759069 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761820c53460443aaf8874c626da6403" candidate_uuid: "7831d89598944a66b3b4d71872d394c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e9f491cf8c4fd9aa59115eac440037" is_pre_election: true
I20260620 19:21:17.759182 21138 raft_consensus.cc:2454] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7831d89598944a66b3b4d71872d394c0 in term 0.
I20260620 19:21:17.759358 20943 leader_election.cc:302] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 581f6b265d7b42c282cca03b716ae6a2, 7831d89598944a66b3b4d71872d394c0; no voters:
I20260620 19:21:17.759496 21204 raft_consensus.cc:2781] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:21:17.759573 21204 raft_consensus.cc:482] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:21:17.759608 21204 raft_consensus.cc:3037] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.760284 21204 raft_consensus.cc:504] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.760414 21204 leader_election.cc:288] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [CANDIDATE]: Term 1 election: Requested vote from peers a4e9f491cf8c4fd9aa59115eac440037 (127.1.1.195:33135), 581f6b265d7b42c282cca03b716ae6a2 (127.1.1.193:34945)
I20260620 19:21:17.760562 21138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761820c53460443aaf8874c626da6403" candidate_uuid: "7831d89598944a66b3b4d71872d394c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e9f491cf8c4fd9aa59115eac440037"
I20260620 19:21:17.760591 20876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761820c53460443aaf8874c626da6403" candidate_uuid: "7831d89598944a66b3b4d71872d394c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "581f6b265d7b42c282cca03b716ae6a2"
I20260620 19:21:17.760654 21138 raft_consensus.cc:3037] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.760654 20876 raft_consensus.cc:3037] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:17.761389 21138 raft_consensus.cc:2454] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7831d89598944a66b3b4d71872d394c0 in term 1.
I20260620 19:21:17.761437 20876 raft_consensus.cc:2454] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7831d89598944a66b3b4d71872d394c0 in term 1.
I20260620 19:21:17.761562 20943 leader_election.cc:302] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7831d89598944a66b3b4d71872d394c0, a4e9f491cf8c4fd9aa59115eac440037; no voters:
I20260620 19:21:17.761639 21204 raft_consensus.cc:2781] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:17.761778 21204 raft_consensus.cc:686] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7831d89598944a66b3b4d71872d394c0, State: Running, Role: LEADER
I20260620 19:21:17.761895 21204 consensus_queue.cc:237] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } }
I20260620 19:21:17.762681 20734 catalog_manager.cc:5495] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7831d89598944a66b3b4d71872d394c0 (127.1.1.194). New cstate: current_term: 1 leader_uuid: "7831d89598944a66b3b4d71872d394c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7831d89598944a66b3b4d71872d394c0" member_type: VOTER last_known_addr { host: "127.1.1.194" port: 35763 } health_report { overall_health: HEALTHY } } }
I20260620 19:21:17.770488 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.768577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.1.1.194:35763)
0620 19:21:17.768613 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:17.768615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:17.768621 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:21:17.768686 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:17.768686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:17.768699 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:17.768744 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:17.768824 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.769271 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.769909 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:17.770069 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:17.770123 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:17.770235 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:17.770309 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:17.770358 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:17.770388 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 19:21:17.770400 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:17.771960 20876 raft_consensus.cc:1261] T 761820c53460443aaf8874c626da6403 P 581f6b265d7b42c282cca03b716ae6a2 [term 1 FOLLOWER]: Refusing update from remote peer 7831d89598944a66b3b4d71872d394c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:17.771960 21138 raft_consensus.cc:1261] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 1 FOLLOWER]: Refusing update from remote peer 7831d89598944a66b3b4d71872d394c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:17.772291 21208 consensus_queue.cc:1035] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e9f491cf8c4fd9aa59115eac440037" member_type: VOTER last_known_addr { host: "127.1.1.195" port: 33135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:17.772413 21204 consensus_queue.cc:1035] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "581f6b265d7b42c282cca03b716ae6a2" member_type: VOTER last_known_addr { host: "127.1.1.193" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:17.774539 21211 mvcc.cc:205] Tried to move back new op lower bound from 7299003505751719936 to 7299003505713258496. Current Snapshot: MvccSnapshot[applied={T|T < 7299003505751719936}]
I20260620 19:21:17.774837 21212 mvcc.cc:205] Tried to move back new op lower bound from 7299003505751719936 to 7299003505713258496. Current Snapshot: MvccSnapshot[applied={T|T < 7299003505751719936}]
I20260620 19:21:17.774873 21213 mvcc.cc:205] Tried to move back new op lower bound from 7299003505751719936 to 7299003505713258496. Current Snapshot: MvccSnapshot[applied={T|T < 7299003505751719936}]
I20260620 19:21:17.776731 20732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:17.776820 20732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:21:17.777621 20732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22772ca92b8b4dfe90446367baf03a3c: Sending DeleteTablet for 3 replicas of tablet 761820c53460443aaf8874c626da6403
I20260620 19:21:17.777977 20987 tablet_service.cc:1514] Processing DeleteTablet for tablet 761820c53460443aaf8874c626da6403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='slave'} at 127.0.0.1:54180
I20260620 19:21:17.778213 21221 tablet_replica.cc:330] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: stopping tablet replica
I20260620 19:21:17.778218 20856 tablet_service.cc:1514] Processing DeleteTablet for tablet 761820c53460443aaf8874c626da6403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='slave'} at 127.0.0.1:47278
I20260620 19:21:17.778313 21221 raft_consensus.cc:2229] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:21:17.778460 21221 raft_consensus.cc:2258] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:17.778451 21118 tablet_service.cc:1514] Processing DeleteTablet for tablet 761820c53460443aaf8874c626da6403 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:21:17 UTC) from {username='slave'} at 127.0.0.1:39950
I20260620 19:21:17.778891 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20794
I20260620 19:21:17.778944 21221 ts_tablet_manager.cc:1905] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:17.780876 21221 ts_tablet_manager.cc:1918] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:17.780951 21221 log.cc:1196] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-1/wal/wals/761820c53460443aaf8874c626da6403
I20260620 19:21:17.781203 21221 ts_tablet_manager.cc:1939] T 761820c53460443aaf8874c626da6403 P 7831d89598944a66b3b4d71872d394c0: Deleting consensus metadata
I20260620 19:21:17.782263 20720 catalog_manager.cc:4841] TS 7831d89598944a66b3b4d71872d394c0 (127.1.1.194:35763): tablet 761820c53460443aaf8874c626da6403 (table test-table [id=4ddd5e2e450643deb4c00ee9b6ffb5fa]) successfully deleted
I20260620 19:21:17.781931 21223 tablet_replica.cc:330] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: stopping tablet replica
I20260620 19:21:17.782438 21223 raft_consensus.cc:2229] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:21:17.782546 21223 raft_consensus.cc:2258] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:21:17.783012 21223 ts_tablet_manager.cc:1905] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:21:17.784555 21223 ts_tablet_manager.cc:1918] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:21:17.784644 21223 log.cc:1196] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Deleting WAL directory at /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781983021660275-1031-0/minicluster-data/ts-2/wal/wals/761820c53460443aaf8874c626da6403
I20260620 19:21:17.784893 21223 ts_tablet_manager.cc:1939] T 761820c53460443aaf8874c626da6403 P a4e9f491cf8c4fd9aa59115eac440037: Deleting consensus metadata
I20260620 19:21:17.785298 20721 catalog_manager.cc:4841] TS a4e9f491cf8c4fd9aa59115eac440037 (127.1.1.195:33135): tablet 761820c53460443aaf8874c626da6403 (table test-table [id=4ddd5e2e450643deb4c00ee9b6ffb5fa]) successfully deleted
W20260620 19:21:17.786495 20721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.1.193:34945 (error 108)
I20260620 19:21:17.786747 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20925
I20260620 19:21:17.786908 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:17.786798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.193:34945 (local address 127.0.0.1:47282)
0620 19:21:17.786841 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:17.786876 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:21:17.787005 20721 catalog_manager.cc:4584] TS 581f6b265d7b42c282cca03b716ae6a2 (127.1.1.193:34945): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 761820c53460443aaf8874c626da6403: Network error: Client connection negotiation failed: client connection to 127.1.1.193:34945: connect: Connection refused (error 111)
I20260620 19:21:17.790755 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21056
I20260620 19:21:17.794669 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 20703
2026-06-20T19:21:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (659 ms)
[----------] 6 tests from EncryptionPolicyTest (7651 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:17 dist-test-slave-mwkb krb5kdc[21227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:17 dist-test-slave-mwkb krb5kdc[21227](info): set up 2 sockets
Jun 20 19:21:17 dist-test-slave-mwkb krb5kdc[21227](info): commencing operation
krb5kdc: starting...
W20260620 19:21:18.740701 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:18 dist-test-slave-mwkb krb5kdc[21227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:18.840793 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:18.909583 21240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:18.909756 21240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:18.909776 21240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:18.910866 21240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:18.910905 21240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:18.910919 21240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:18.910969 21240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:18.910995 21240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:18.912504 21240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:18.912736 21240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:18.912993 21240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:18.917316 21245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:18.917368 21246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:18.917326 21248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:18.917397 21240 server_base.cc:1034] running on GCE node
I20260620 19:21:18.917832 21240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:18.918083 21240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:18.919232 21240 hybrid_clock.cc:648] HybridClock initialized: now 1781983278919225 us; error 33 us; skew 500 ppm
Jun 20 19:21:18 dist-test-slave-mwkb krb5kdc[21227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:18.921552 21240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:18.921913 21240 webserver.cc:460] Webserver started at http://127.0.0.1:42719/ using document root <none> and password file <none>
I20260620 19:21:18.922120 21240 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:18.922181 21240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:18.922313 21240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:18.923265 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "e090dbe647b04e7c9684ad4d95105aa6"
format_stamp: "Formatted at 2026-06-20 19:21:18 on dist-test-slave-mwkb"
server_key: "3c0e1d44838777e51719b74345c0cbb3"
server_key_iv: "9704df401c33e23d7144197f81c3b7e2"
server_key_version: "encryptionkey@0"
I20260620 19:21:18.923609 21240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "e090dbe647b04e7c9684ad4d95105aa6"
format_stamp: "Formatted at 2026-06-20 19:21:18 on dist-test-slave-mwkb"
server_key: "3c0e1d44838777e51719b74345c0cbb3"
server_key_iv: "9704df401c33e23d7144197f81c3b7e2"
server_key_version: "encryptionkey@0"
I20260620 19:21:18.924768 21240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:18.925436 21255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:18.925580 21240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:18.925666 21240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "e090dbe647b04e7c9684ad4d95105aa6"
format_stamp: "Formatted at 2026-06-20 19:21:18 on dist-test-slave-mwkb"
server_key: "3c0e1d44838777e51719b74345c0cbb3"
server_key_iv: "9704df401c33e23d7144197f81c3b7e2"
server_key_version: "encryptionkey@0"
I20260620 19:21:18.925732 21240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:18.949290 21240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:18.951175 21240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:18.951311 21240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:18.955024 21240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36505
I20260620 19:21:18.955054 21307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36505 every 8 connection(s)
I20260620 19:21:18.955353 21240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:18.956194 21308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:18.958019 21308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6: Bootstrap starting.
I20260620 19:21:18.958489 21308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:18.958740 21308 log.cc:824] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:18.959321 21308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6: No bootstrap required, opened a new log
I20260620 19:21:18.960763 21308 raft_consensus.cc:348] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260620 19:21:18.960884 21308 raft_consensus.cc:374] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:18.960937 21308 raft_consensus.cc:729] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e090dbe647b04e7c9684ad4d95105aa6, State: Initialized, Role: FOLLOWER
I20260620 19:21:18.961063 21308 consensus_queue.cc:260] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260620 19:21:18.961138 21308 raft_consensus.cc:388] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:18.961189 21308 raft_consensus.cc:482] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:18.961231 21308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:18.961695 21308 raft_consensus.cc:504] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260620 19:21:18.961791 21308 leader_election.cc:302] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e090dbe647b04e7c9684ad4d95105aa6; no voters:
I20260620 19:21:18.961966 21308 leader_election.cc:288] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:18.962013 21311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:18.962168 21311 raft_consensus.cc:686] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [term 1 LEADER]: Becoming Leader. State: Replica: e090dbe647b04e7c9684ad4d95105aa6, State: Running, Role: LEADER
I20260620 19:21:18.962184 21308 sys_catalog.cc:564] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:18.962280 21311 consensus_queue.cc:237] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } }
I20260620 19:21:18.962594 21313 sys_catalog.cc:455] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e090dbe647b04e7c9684ad4d95105aa6. Latest consensus state: current_term: 1 leader_uuid: "e090dbe647b04e7c9684ad4d95105aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } } }
I20260620 19:21:18.962692 21312 sys_catalog.cc:455] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e090dbe647b04e7c9684ad4d95105aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e090dbe647b04e7c9684ad4d95105aa6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36505 } } }
I20260620 19:21:18.962744 21313 sys_catalog.cc:458] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:18.962800 21312 sys_catalog.cc:458] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:18.963143 21320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:18.963650 21320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:18.964570 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21240
I20260620 19:21:18.964668 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:18.964834 1031 external_mini_cluster.cc:1428] Setting key 1624376ea9ad5dcf3d339d696feae199
I20260620 19:21:18.965191 21320 catalog_manager.cc:1347] Generated new cluster ID: 803ea4057b7140789d875c9d36493f45
I20260620 19:21:18.965271 21320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:18 dist-test-slave-mwkb krb5kdc[21227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:21:18.974783 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:18.965233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35699 (local address 127.0.0.1:36505)
0620 19:21:18.965510 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:18.965517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:18.965554 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:21:18.965643 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:18.965648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:18.965691 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:18.965808 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:18.966406 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:18.967622 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:18.968564 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:18.969133 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:18.969231 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:18.971530 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:18.971534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:18.971534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:18.971543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:18.973231 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:18.973663 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:18.973666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:18.973666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:18.973757 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:18.974001 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:18.974001 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:18.974002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:18.974243 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:18.974384 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:18.974504 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:21:18.974600 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":67,"threads_started":1}
I20260620 19:21:18.975654 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:18.983202 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:18.976918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39889 (local address 127.0.0.1:36505)
0620 19:21:18.976972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:18.976973 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:18.977026 (+ 53us) server_negotiation.cc:407] Connection header received
0620 19:21:18.977124 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:18.977125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:18.977132 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:18.977183 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:18.977742 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:18.978390 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:18.979261 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:18.979559 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:18.979612 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:18.980883 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:18.980884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:18.980884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:18.980889 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:18.982133 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:18.982488 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:18.982488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:18.982489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:18.982532 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:18.982723 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:18.982724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:18.982724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:18.982834 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:18.982924 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:18.983070 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:21:18.983126 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:21:18.999133 21320 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:18.999591 21320 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:19.004638 21320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e090dbe647b04e7c9684ad4d95105aa6: Generated new TSK 0
I20260620 19:21:19.004782 21320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:19.012859 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21240
W20260620 19:21:19.016670 21225 connection.cc:535] client connection to 127.0.0.1:36505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36505): Connection reset by peer (error 104)
2026-06-20T19:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:19 dist-test-slave-mwkb krb5kdc[21339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:19 dist-test-slave-mwkb krb5kdc[21339](info): set up 2 sockets
Jun 20 19:21:19 dist-test-slave-mwkb krb5kdc[21339](info): commencing operation
krb5kdc: starting...
W20260620 19:21:19.960429 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:20 dist-test-slave-mwkb krb5kdc[21339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:20.059435 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:20.130594 21352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:20.130774 21352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:20.130795 21352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:20.131855 21352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:20.131886 21352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:20.131898 21352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:20.131909 21352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:20.131920 21352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:20.133239 21352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:20.133409 21352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:20.133558 21352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:20.137874 21358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:20.137957 21357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:20.137871 21360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:20.138170 21352 server_base.cc:1034] running on GCE node
I20260620 19:21:20.138414 21352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:20.138682 21352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:20.139834 21352 hybrid_clock.cc:648] HybridClock initialized: now 1781983280139825 us; error 36 us; skew 500 ppm
Jun 20 19:21:20 dist-test-slave-mwkb krb5kdc[21339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:20.144670 21352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:20.145031 21352 webserver.cc:460] Webserver started at http://127.0.0.1:34913/ using document root <none> and password file <none>
I20260620 19:21:20.145283 21352 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:20.145360 21352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:20.145506 21352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:20.146389 21352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "e395acdff37c44d3820cfcec1414e853"
format_stamp: "Formatted at 2026-06-20 19:21:20 on dist-test-slave-mwkb"
server_key: "3e72ca8f307666b3d22fbdbeb8110253"
server_key_iv: "0f382e69edf2f4c49ae8136762adb837"
server_key_version: "encryptionkey@0"
I20260620 19:21:20.146713 21352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "e395acdff37c44d3820cfcec1414e853"
format_stamp: "Formatted at 2026-06-20 19:21:20 on dist-test-slave-mwkb"
server_key: "3e72ca8f307666b3d22fbdbeb8110253"
server_key_iv: "0f382e69edf2f4c49ae8136762adb837"
server_key_version: "encryptionkey@0"
I20260620 19:21:20.147887 21352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:20.148536 21367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:20.148689 21352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:20.148783 21352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "e395acdff37c44d3820cfcec1414e853"
format_stamp: "Formatted at 2026-06-20 19:21:20 on dist-test-slave-mwkb"
server_key: "3e72ca8f307666b3d22fbdbeb8110253"
server_key_iv: "0f382e69edf2f4c49ae8136762adb837"
server_key_version: "encryptionkey@0"
I20260620 19:21:20.148855 21352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:20.167141 21352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:20.168671 21352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:20.168812 21352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:20.172422 21352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40537
I20260620 19:21:20.172454 21419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40537 every 8 connection(s)
I20260620 19:21:20.172744 21352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:20.173280 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21352
I20260620 19:21:20.173393 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:20.173560 1031 external_mini_cluster.cc:1428] Setting key 1458e0a51a5c4c99f8059794923b2879
I20260620 19:21:20.173761 21420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:20.176244 21420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853: Bootstrap starting.
I20260620 19:21:20.176748 21420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:20.177006 21420 log.cc:824] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:20.177712 21420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853: No bootstrap required, opened a new log
I20260620 19:21:20.178498 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:20.174014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57829 (local address 127.0.0.1:40537)
0620 19:21:20.174215 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:20.174222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:20.174254 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:21:20.174690 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:20.174695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:20.174733 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:20.174857 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:20.175088 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:20.175920 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:20.176740 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:20.177215 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:20.177333 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:20.177948 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:20.177952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:20.177952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:20.178096 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:20.178120 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:20.178129 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:20.178232 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:20.178308 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:21:20.178333 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 19:21:20.179489 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:20.179814 21420 raft_consensus.cc:348] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260620 19:21:20.179960 21420 raft_consensus.cc:374] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:20.180006 21420 raft_consensus.cc:729] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e395acdff37c44d3820cfcec1414e853, State: Initialized, Role: FOLLOWER
I20260620 19:21:20.180120 21420 consensus_queue.cc:260] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260620 19:21:20.180186 21420 raft_consensus.cc:388] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:20.180205 21420 raft_consensus.cc:482] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:20.180269 21420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:20.180898 21420 raft_consensus.cc:504] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260620 19:21:20.180974 21420 leader_election.cc:302] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e395acdff37c44d3820cfcec1414e853; no voters:
I20260620 19:21:20.181115 21420 leader_election.cc:288] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:20.181133 21430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:20.181293 21430 raft_consensus.cc:686] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [term 1 LEADER]: Becoming Leader. State: Replica: e395acdff37c44d3820cfcec1414e853, State: Running, Role: LEADER
I20260620 19:21:20.181326 21420 sys_catalog.cc:564] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:20.181414 21430 consensus_queue.cc:237] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260620 19:21:20.181828 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e395acdff37c44d3820cfcec1414e853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260620 19:21:20.181910 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:20.181830 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e395acdff37c44d3820cfcec1414e853. Latest consensus state: current_term: 1 leader_uuid: "e395acdff37c44d3820cfcec1414e853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e395acdff37c44d3820cfcec1414e853" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260620 19:21:20.182154 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:20.182399 21436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:20.182860 21436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:20.184350 21436 catalog_manager.cc:1347] Generated new cluster ID: 66da3e0bdf0e4f07a0e436b3a7838f9b
I20260620 19:21:20.184437 21436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:20.184508 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:20.180712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41557 (local address 127.0.0.1:40537)
0620 19:21:20.180740 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:20.180741 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:20.181230 (+ 489us) server_negotiation.cc:407] Connection header received
0620 19:21:20.181315 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:20.181315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:20.181325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:20.181377 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:20.181576 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:20.182197 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:20.183230 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:20.183530 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:20.183570 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:20.184281 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:20.184282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:20.184283 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:20.184286 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:20.184297 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:20.184301 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:20.184345 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:20.184444 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 19:21:20.184454 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:21:20.192368 21436 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:20.192775 21436 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:20.197942 21436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e395acdff37c44d3820cfcec1414e853: Generated new TSK 0
I20260620 19:21:20.198096 21436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:20.202720 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21352
W20260620 19:21:20.206182 21337 connection.cc:535] client connection to 127.0.0.1:40537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40537): Connection reset by peer (error 104)
2026-06-20T19:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:20 dist-test-slave-mwkb krb5kdc[21451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:20 dist-test-slave-mwkb krb5kdc[21451](info): set up 2 sockets
Jun 20 19:21:20 dist-test-slave-mwkb krb5kdc[21451](info): commencing operation
krb5kdc: starting...
W20260620 19:21:21.140794 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:21 dist-test-slave-mwkb krb5kdc[21451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:21.243454 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:21.313668 21464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:21.313838 21464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:21.313859 21464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:21.314903 21464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:21.314937 21464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:21.314951 21464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:21.314965 21464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:21.314975 21464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:21.316272 21464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41611
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:21.316416 21464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:21.316562 21464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:21.320899 21469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:21.320889 21470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:21.320968 21472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:21.321022 21464 server_base.cc:1034] running on GCE node
I20260620 19:21:21.321368 21464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:21.321581 21464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:21.322710 21464 hybrid_clock.cc:648] HybridClock initialized: now 1781983281322699 us; error 26 us; skew 500 ppm
Jun 20 19:21:21 dist-test-slave-mwkb krb5kdc[21451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:21.324913 21464 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:21.325289 21464 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260620 19:21:21.325451 21464 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:21.325486 21464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:21.325559 21464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:21.326359 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3"
format_stamp: "Formatted at 2026-06-20 19:21:21 on dist-test-slave-mwkb"
server_key: "38447497361a5761ede78cef2d4d9203"
server_key_iv: "4430bdf0f0b11fe56a1035e1d3146d4c"
server_key_version: "encryptionkey@0"
I20260620 19:21:21.326609 21464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3"
format_stamp: "Formatted at 2026-06-20 19:21:21 on dist-test-slave-mwkb"
server_key: "38447497361a5761ede78cef2d4d9203"
server_key_iv: "4430bdf0f0b11fe56a1035e1d3146d4c"
server_key_version: "encryptionkey@0"
I20260620 19:21:21.327769 21464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:21.328368 21479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:21.328534 21464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:21.328631 21464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3"
format_stamp: "Formatted at 2026-06-20 19:21:21 on dist-test-slave-mwkb"
server_key: "38447497361a5761ede78cef2d4d9203"
server_key_iv: "4430bdf0f0b11fe56a1035e1d3146d4c"
server_key_version: "encryptionkey@0"
I20260620 19:21:21.328706 21464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:21.331655 21464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:21.333654 21464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:21.333801 21464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:21.337235 21464 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41611
I20260620 19:21:21.337260 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41611 every 8 connection(s)
I20260620 19:21:21.337517 21464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:21.338308 21532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:21.340093 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3: Bootstrap starting.
I20260620 19:21:21.340612 21532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:21.340858 21532 log.cc:824] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:21.341464 21532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3: No bootstrap required, opened a new log
I20260620 19:21:21.342794 21532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } }
I20260620 19:21:21.342904 21532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:21.342955 21532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5b2f44b3a54b2bbb7b41aba9acf9f3, State: Initialized, Role: FOLLOWER
I20260620 19:21:21.343150 21532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } }
I20260620 19:21:21.343232 21532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:21.343279 21532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:21.343327 21532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:21.343807 21532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } }
I20260620 19:21:21.343911 21532 leader_election.cc:302] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c5b2f44b3a54b2bbb7b41aba9acf9f3; no voters:
I20260620 19:21:21.344084 21532 leader_election.cc:288] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:21.344133 21535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:21.344310 21535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4c5b2f44b3a54b2bbb7b41aba9acf9f3, State: Running, Role: LEADER
I20260620 19:21:21.344317 21532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:21.344393 21535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } }
I20260620 19:21:21.344731 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } } }
I20260620 19:21:21.344826 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:21.344733 21537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c5b2f44b3a54b2bbb7b41aba9acf9f3. Latest consensus state: current_term: 1 leader_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c5b2f44b3a54b2bbb7b41aba9acf9f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41611 } } }
I20260620 19:21:21.344945 21537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:21.345047 21539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:21.345549 21539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:21.346951 21539 catalog_manager.cc:1347] Generated new cluster ID: ce58c90071b1456da5a3ac20990e4c56
I20260620 19:21:21.347012 21539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:21.347084 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21464
I20260620 19:21:21.347179 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:21.347330 1031 external_mini_cluster.cc:1428] Setting key 126e5ebd1c307d4bc7cda6c50767b829
I20260620 19:21:21.349210 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:21.347665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37759 (local address 127.0.0.1:41611)
0620 19:21:21.347946 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:21.347953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:21.347980 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:21:21.348084 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:21.348101 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:21.348126 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:21.348242 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:21.348732 (+ 490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:21.348735 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:21.348735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:21.348739 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:21.348771 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:21.348784 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:21.348841 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:21.348898 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:21:21.349013 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":69,"threads_started":1}
I20260620 19:21:21.350018 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:21.352468 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:21.351400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53489 (local address 127.0.0.1:41611)
0620 19:21:21.351453 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:21.351455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:21.351648 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:21:21.351692 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:21.351693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:21.351698 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:21.351753 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:21.352257 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:21.352258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:21.352258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:21.352262 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:21.352274 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:21.352278 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:21.352348 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:21.352409 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:21:21.352419 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:21:21.353116 21539 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:21.353247 21539 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:21.357607 21539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c5b2f44b3a54b2bbb7b41aba9acf9f3: Generated new TSK 0
I20260620 19:21:21.357738 21539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:21.359151 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21464
2026-06-20T19:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:21 dist-test-slave-mwkb krb5kdc[21563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:21 dist-test-slave-mwkb krb5kdc[21563](info): set up 2 sockets
Jun 20 19:21:21 dist-test-slave-mwkb krb5kdc[21563](info): commencing operation
krb5kdc: starting...
W20260620 19:21:22.308553 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:22.407361 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:22.479113 21576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:22.479285 21576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:22.479317 21576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:22.480445 21576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:22.480513 21576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:22.480538 21576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:22.480569 21576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:22.480592 21576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:22.481925 21576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:22.482173 21576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:22.482371 21576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:22.486582 21581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:22.486591 21582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:22.486591 21584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:22.486739 21576 server_base.cc:1034] running on GCE node
I20260620 19:21:22.486987 21576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:22.487231 21576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:22.488355 21576 hybrid_clock.cc:648] HybridClock initialized: now 1781983282488342 us; error 30 us; skew 500 ppm
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983282, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:22.490455 21576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:22.490892 21576 webserver.cc:460] Webserver started at http://127.0.0.1:35849/ using document root <none> and password file <none>
I20260620 19:21:22.491127 21576 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:22.491168 21576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:22.491325 21576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:22.492179 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "9faeb8ea82434fe8b11c35db80c69ce9"
format_stamp: "Formatted at 2026-06-20 19:21:22 on dist-test-slave-mwkb"
server_key: "0bdc63283edee1b6fd85550dc2d7ceed"
server_key_iv: "512f1b26df933a939c3ef9cb098fb7a3"
server_key_version: "encryptionkey@0"
I20260620 19:21:22.492476 21576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "9faeb8ea82434fe8b11c35db80c69ce9"
format_stamp: "Formatted at 2026-06-20 19:21:22 on dist-test-slave-mwkb"
server_key: "0bdc63283edee1b6fd85550dc2d7ceed"
server_key_iv: "512f1b26df933a939c3ef9cb098fb7a3"
server_key_version: "encryptionkey@0"
I20260620 19:21:22.493728 21576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:22.494470 21591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:22.494648 21576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:22.494728 21576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "9faeb8ea82434fe8b11c35db80c69ce9"
format_stamp: "Formatted at 2026-06-20 19:21:22 on dist-test-slave-mwkb"
server_key: "0bdc63283edee1b6fd85550dc2d7ceed"
server_key_iv: "512f1b26df933a939c3ef9cb098fb7a3"
server_key_version: "encryptionkey@0"
I20260620 19:21:22.494794 21576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:22.506531 21576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:22.507998 21576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:22.508140 21576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:22.511982 21576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33051
I20260620 19:21:22.512032 21643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33051 every 8 connection(s)
I20260620 19:21:22.512311 21576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:22.513160 21644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:22.515113 21644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9: Bootstrap starting.
I20260620 19:21:22.515656 21644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:22.515914 21644 log.cc:824] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:22.516498 21644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9: No bootstrap required, opened a new log
I20260620 19:21:22.517979 21644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } }
I20260620 19:21:22.518115 21644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:22.518155 21644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9faeb8ea82434fe8b11c35db80c69ce9, State: Initialized, Role: FOLLOWER
I20260620 19:21:22.518301 21644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } }
I20260620 19:21:22.518354 21644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:22.518389 21644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:22.518433 21644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:22.518875 21644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } }
I20260620 19:21:22.518972 21644 leader_election.cc:302] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9faeb8ea82434fe8b11c35db80c69ce9; no voters:
I20260620 19:21:22.519145 21644 leader_election.cc:288] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:22.519176 21647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:22.519363 21647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 9faeb8ea82434fe8b11c35db80c69ce9, State: Running, Role: LEADER
I20260620 19:21:22.519418 21644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:22.519497 21647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } }
I20260620 19:21:22.519893 21648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } } }
I20260620 19:21:22.519902 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9faeb8ea82434fe8b11c35db80c69ce9. Latest consensus state: current_term: 1 leader_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9faeb8ea82434fe8b11c35db80c69ce9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33051 } } }
I20260620 19:21:22.520052 21648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:22.520067 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:22.520431 21651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:22.520890 21651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:22.521049 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21576
I20260620 19:21:22.521186 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:22.521358 1031 external_mini_cluster.cc:1428] Setting key 21f6490214f4cb9cd7af7f27e8fde4c7
I20260620 19:21:22.522742 21651 catalog_manager.cc:1347] Generated new cluster ID: 3a164a0112c4462b8860008fbf48953b
I20260620 19:21:22.522790 21651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:21:22.530201 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:22.521670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53065 (local address 127.0.0.1:33051)
0620 19:21:22.522000 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:22.522006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:22.522024 (+ 18us) server_negotiation.cc:407] Connection header received
0620 19:21:22.522145 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:22.522151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:22.522199 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:22.522330 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:22.522992 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:22.523894 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:22.524644 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:22.525002 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:22.525118 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:22.527172 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:22.527177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:22.527177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:22.527187 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:22.528754 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:22.529156 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:22.529159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:22.529159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:22.529249 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:22.529456 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:22.529457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:22.529457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:22.529686 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:22.529846 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:22.529903 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:21:22.530025 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":80,"threads_started":1}
I20260620 19:21:22.531033 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:22.537952 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:22.532238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50563 (local address 127.0.0.1:33051)
0620 19:21:22.532265 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:22.532266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:22.532359 (+ 93us) server_negotiation.cc:407] Connection header received
0620 19:21:22.532406 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:22.532407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:22.532415 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:22.532451 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:22.532892 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:22.533427 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:22.534157 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:22.534387 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:22.534466 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:22.535506 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:22.535507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:22.535507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:22.535513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:22.536851 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:22.537268 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:22.537269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:22.537269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:22.537314 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:22.537495 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:22.537496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:22.537496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:22.537613 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:22.537700 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:22.537802 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:21:22.537866 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:21:22.539126 21651 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:22.539572 21651 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:22.543648 21651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9faeb8ea82434fe8b11c35db80c69ce9: Generated new TSK 0
I20260620 19:21:22.543788 21651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:22.548522 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21576
2026-06-20T19:21:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21675](info): set up 2 sockets
Jun 20 19:21:22 dist-test-slave-mwkb krb5kdc[21675](info): commencing operation
krb5kdc: starting...
W20260620 19:21:23.488605 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:23 dist-test-slave-mwkb krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:23.591552 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:23.668282 21688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:23.668483 21688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:23.668532 21688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:23.669651 21688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:23.669713 21688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:23.669737 21688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:23.669792 21688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:23.669819 21688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:23.671205 21688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:23.671448 21688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:23.671663 21688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:23.676120 21694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:23.676175 21688 server_base.cc:1034] running on GCE node
W20260620 19:21:23.676128 21696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:23.676257 21693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:23.676528 21688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:23.676743 21688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:23.677875 21688 hybrid_clock.cc:648] HybridClock initialized: now 1781983283677857 us; error 36 us; skew 500 ppm
Jun 20 19:21:23 dist-test-slave-mwkb krb5kdc[21675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:23.680059 21688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:23.680438 21688 webserver.cc:460] Webserver started at http://127.0.0.1:35263/ using document root <none> and password file <none>
I20260620 19:21:23.680675 21688 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:23.680720 21688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:23.680863 21688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:23.681855 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "50abdfba1704434fbc9e1f7ffe37dbe7"
format_stamp: "Formatted at 2026-06-20 19:21:23 on dist-test-slave-mwkb"
server_key: "a756332525f34186ebefb962099069c9"
server_key_iv: "f2c4c0ee2161803b764ba466903e2f2e"
server_key_version: "encryptionkey@0"
I20260620 19:21:23.682188 21688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "50abdfba1704434fbc9e1f7ffe37dbe7"
format_stamp: "Formatted at 2026-06-20 19:21:23 on dist-test-slave-mwkb"
server_key: "a756332525f34186ebefb962099069c9"
server_key_iv: "f2c4c0ee2161803b764ba466903e2f2e"
server_key_version: "encryptionkey@0"
I20260620 19:21:23.683398 21688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:21:23.684041 21703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:23.684201 21688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:23.684285 21688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "50abdfba1704434fbc9e1f7ffe37dbe7"
format_stamp: "Formatted at 2026-06-20 19:21:23 on dist-test-slave-mwkb"
server_key: "a756332525f34186ebefb962099069c9"
server_key_iv: "f2c4c0ee2161803b764ba466903e2f2e"
server_key_version: "encryptionkey@0"
I20260620 19:21:23.684369 21688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:23.706758 21688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:23.708336 21688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:23.708485 21688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:23.712049 21688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46217
I20260620 19:21:23.712081 21755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46217 every 8 connection(s)
I20260620 19:21:23.712375 21688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:23.713277 21756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:23.715174 21756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: Bootstrap starting.
I20260620 19:21:23.715291 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21688
I20260620 19:21:23.715401 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:23.715583 1031 external_mini_cluster.cc:1428] Setting key 8d7c190f0fd96bacc1c5934823ba43e3
I20260620 19:21:23.715809 21756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:23.716142 21756 log.cc:824] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:23.716836 21756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: No bootstrap required, opened a new log
I20260620 19:21:23.719106 21756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260620 19:21:23.719220 21756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:23.719241 21756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50abdfba1704434fbc9e1f7ffe37dbe7, State: Initialized, Role: FOLLOWER
I20260620 19:21:23.719373 21756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260620 19:21:23.719446 21756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:23.719468 21756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:23.719540 21756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:23.720018 21756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260620 19:21:23.720119 21756 leader_election.cc:302] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50abdfba1704434fbc9e1f7ffe37dbe7; no voters:
I20260620 19:21:23.720268 21756 leader_election.cc:288] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:23.720296 21761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:23.720415 21761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [term 1 LEADER]: Becoming Leader. State: Replica: 50abdfba1704434fbc9e1f7ffe37dbe7, State: Running, Role: LEADER
I20260620 19:21:23.720496 21761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } }
I20260620 19:21:23.720806 21756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:23.721520 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:23.716019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55799 (local address 127.0.0.1:46217)
0620 19:21:23.716211 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:23.716219 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:23.716248 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:23.716762 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:23.716770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:23.716825 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:23.716969 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:23.717307 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:23.718704 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:23.719594 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:23.719983 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:23.720118 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:23.720787 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:23.720791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:23.720791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:23.720975 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:23.721010 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:23.721024 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:23.721144 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:23.721240 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:21:23.721320 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260620 19:21:23.722323 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } } }
I20260620 19:21:23.722407 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:23.722494 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:23.720790 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50abdfba1704434fbc9e1f7ffe37dbe7. Latest consensus state: current_term: 1 leader_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50abdfba1704434fbc9e1f7ffe37dbe7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46217 } } }
I20260620 19:21:23.722569 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7 [sys.catalog]: This master's current role is: LEADER
W20260620 19:21:23.722690 21776 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:23.722718 21776 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:23.722786 21778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:23.722915 21778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:23.724152 21778 catalog_manager.cc:1347] Generated new cluster ID: 5778aee9f7e84baeb167ff902491c882
I20260620 19:21:23.724200 21778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:23.728247 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:23.724007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46957 (local address 127.0.0.1:46217)
0620 19:21:23.724044 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:23.724046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:23.725160 (+ 1114us) server_negotiation.cc:407] Connection header received
0620 19:21:23.725179 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:23.725180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:23.725189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:23.725250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:23.725438 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:23.726134 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:23.727130 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:23.727409 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:23.727462 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:23.728044 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:23.728045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:23.728046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:23.728049 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:23.728060 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:23.728064 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:23.728108 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:23.728169 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 19:21:23.728186 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:23.753834 21778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:23.754338 21778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:23.757417 21778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50abdfba1704434fbc9e1f7ffe37dbe7: Generated new TSK 0
I20260620 19:21:23.757567 21778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:23.767235 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21688
2026-06-20T19:21:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:23 dist-test-slave-mwkb krb5kdc[21787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:23 dist-test-slave-mwkb krb5kdc[21787](info): set up 2 sockets
Jun 20 19:21:23 dist-test-slave-mwkb krb5kdc[21787](info): commencing operation
krb5kdc: starting...
W20260620 19:21:24.740639 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:24 dist-test-slave-mwkb krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:24.838004 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:24.913746 21800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:24.913931 21800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:24.913954 21800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:24.915089 21800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:24.915130 21800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:24.915144 21800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:24.915158 21800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:24.915205 21800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:24.916622 21800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:24.916857 21800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:24.917088 21800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:21:24.921641 21800 server_base.cc:1034] running on GCE node
W20260620 19:21:24.921605 21805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:24.921604 21806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:24.921615 21808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:24.921983 21800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:24.922304 21800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:24.923454 21800 hybrid_clock.cc:648] HybridClock initialized: now 1781983284923442 us; error 28 us; skew 500 ppm
Jun 20 19:21:24 dist-test-slave-mwkb krb5kdc[21787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983284, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:24.925745 21800 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:24.926183 21800 webserver.cc:460] Webserver started at http://127.0.0.1:41913/ using document root <none> and password file <none>
I20260620 19:21:24.926407 21800 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:24.926458 21800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:24.926599 21800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:24.927560 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "d641ae0fb5f040df8dbd6884ea46721c"
format_stamp: "Formatted at 2026-06-20 19:21:24 on dist-test-slave-mwkb"
server_key: "fcdc80f64721acaab53ccf88216c8bb8"
server_key_iv: "fae4ec5b596c6d8572b1268c3c22d190"
server_key_version: "encryptionkey@0"
I20260620 19:21:24.927892 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "d641ae0fb5f040df8dbd6884ea46721c"
format_stamp: "Formatted at 2026-06-20 19:21:24 on dist-test-slave-mwkb"
server_key: "fcdc80f64721acaab53ccf88216c8bb8"
server_key_iv: "fae4ec5b596c6d8572b1268c3c22d190"
server_key_version: "encryptionkey@0"
I20260620 19:21:24.929191 21800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:24.929906 21815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:24.930086 21800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:24.930174 21800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d641ae0fb5f040df8dbd6884ea46721c"
format_stamp: "Formatted at 2026-06-20 19:21:24 on dist-test-slave-mwkb"
server_key: "fcdc80f64721acaab53ccf88216c8bb8"
server_key_iv: "fae4ec5b596c6d8572b1268c3c22d190"
server_key_version: "encryptionkey@0"
I20260620 19:21:24.930267 21800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:24.933364 21800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:24.935035 21800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:24.935185 21800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:24.939239 21800 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44949
I20260620 19:21:24.939267 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44949 every 8 connection(s)
I20260620 19:21:24.939579 21800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:24.940436 21868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:24.941630 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21800
I20260620 19:21:24.941795 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:24.941984 1031 external_mini_cluster.cc:1428] Setting key d6f6aadc6d0b86809f16e5a20b46a192
I20260620 19:21:24.942970 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c: Bootstrap starting.
I20260620 19:21:24.943652 21868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:24.944001 21868 log.cc:824] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:24.944224 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:24.942427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58189 (local address 127.0.0.1:44949)
0620 19:21:24.942604 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:24.942611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:24.942633 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:21:24.942989 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:24.942994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:24.943043 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:24.943166 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:24.943784 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:24.943794 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:24.943794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:24.943798 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:24.943830 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:24.943839 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:24.943904 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:24.943982 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:21:24.944030 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260620 19:21:24.944746 21868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c: No bootstrap required, opened a new log
I20260620 19:21:24.945129 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:24.946774 21868 raft_consensus.cc:348] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260620 19:21:24.946946 21868 raft_consensus.cc:374] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:24.946985 21868 raft_consensus.cc:729] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d641ae0fb5f040df8dbd6884ea46721c, State: Initialized, Role: FOLLOWER
I20260620 19:21:24.947106 21868 consensus_queue.cc:260] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260620 19:21:24.947221 21868 raft_consensus.cc:388] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:24.947266 21868 raft_consensus.cc:482] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:24.947317 21868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:24.947610 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:24.946562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51083 (local address 127.0.0.1:44949)
0620 19:21:24.946601 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:24.946603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:24.946694 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:21:24.946749 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:24.946750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:24.946753 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:24.946797 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:24.947382 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:24.947383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:24.947384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:24.947389 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:24.947401 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:24.947405 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:24.947468 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:24.947532 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:21:24.947542 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:21:24.947961 21868 raft_consensus.cc:504] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260620 19:21:24.948086 21868 leader_election.cc:302] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d641ae0fb5f040df8dbd6884ea46721c; no voters:
I20260620 19:21:24.948247 21868 leader_election.cc:288] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:24.948290 21878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:24.948475 21868 sys_catalog.cc:564] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:24.948475 21878 raft_consensus.cc:686] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [term 1 LEADER]: Becoming Leader. State: Replica: d641ae0fb5f040df8dbd6884ea46721c, State: Running, Role: LEADER
I20260620 19:21:24.948577 21878 consensus_queue.cc:237] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } }
I20260620 19:21:24.948901 21879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d641ae0fb5f040df8dbd6884ea46721c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260620 19:21:24.948908 21880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d641ae0fb5f040df8dbd6884ea46721c. Latest consensus state: current_term: 1 leader_uuid: "d641ae0fb5f040df8dbd6884ea46721c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d641ae0fb5f040df8dbd6884ea46721c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44949 } } }
I20260620 19:21:24.949036 21879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:24.949159 21880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:24.949319 21887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:24.949790 21887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:24.951238 21887 catalog_manager.cc:1347] Generated new cluster ID: 7065f69273bc4f8985a749a11ee09594
I20260620 19:21:24.951278 21887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:24.966701 21887 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:24.966856 21887 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:24.968997 21887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d641ae0fb5f040df8dbd6884ea46721c: Generated new TSK 0
I20260620 19:21:24.969177 21887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:24.977813 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21800
2026-06-20T19:21:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:25 dist-test-slave-mwkb krb5kdc[21899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:25 dist-test-slave-mwkb krb5kdc[21899](info): set up 2 sockets
Jun 20 19:21:25 dist-test-slave-mwkb krb5kdc[21899](info): commencing operation
krb5kdc: starting...
W20260620 19:21:25.932789 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:25 dist-test-slave-mwkb krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:26.033452 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:26.105407 21912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:26.105672 21912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:26.105695 21912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:26.106827 21912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:26.106869 21912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:26.106885 21912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:26.106935 21912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:26.106953 21912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:26.108405 21912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:26.108639 21912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:26.108841 21912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:26.113348 21917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:26.113406 21920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:26.113440 21918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:26.113765 21912 server_base.cc:1034] running on GCE node
I20260620 19:21:26.113996 21912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:26.114274 21912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:26.115443 21912 hybrid_clock.cc:648] HybridClock initialized: now 1781983286115436 us; error 34 us; skew 500 ppm
Jun 20 19:21:26 dist-test-slave-mwkb krb5kdc[21899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983286, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:26.124033 21912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:26.124460 21912 webserver.cc:460] Webserver started at http://127.0.0.1:35467/ using document root <none> and password file <none>
I20260620 19:21:26.124672 21912 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:26.124737 21912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:26.124873 21912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:26.125859 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "688b466c3a6140a8938d5462eea364da"
format_stamp: "Formatted at 2026-06-20 19:21:26 on dist-test-slave-mwkb"
server_key: "40846bc16e69c1c6a3dffbe2eb64b054"
server_key_iv: "8438fbc4451c853f2c79911f78b3fe30"
server_key_version: "encryptionkey@0"
I20260620 19:21:26.126197 21912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "688b466c3a6140a8938d5462eea364da"
format_stamp: "Formatted at 2026-06-20 19:21:26 on dist-test-slave-mwkb"
server_key: "40846bc16e69c1c6a3dffbe2eb64b054"
server_key_iv: "8438fbc4451c853f2c79911f78b3fe30"
server_key_version: "encryptionkey@0"
I20260620 19:21:26.127404 21912 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:26.128052 21927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:26.128218 21912 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:26.128304 21912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "688b466c3a6140a8938d5462eea364da"
format_stamp: "Formatted at 2026-06-20 19:21:26 on dist-test-slave-mwkb"
server_key: "40846bc16e69c1c6a3dffbe2eb64b054"
server_key_iv: "8438fbc4451c853f2c79911f78b3fe30"
server_key_version: "encryptionkey@0"
I20260620 19:21:26.128377 21912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:26.134398 21912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:26.135965 21912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:26.136094 21912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:26.139765 21912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39203
I20260620 19:21:26.139793 21979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39203 every 8 connection(s)
I20260620 19:21:26.140100 21912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:26.140969 21980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:26.142920 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: Bootstrap starting.
I20260620 19:21:26.143378 21980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:26.143632 21980 log.cc:824] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:26.144244 21980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: No bootstrap required, opened a new log
I20260620 19:21:26.145668 21980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260620 19:21:26.145777 21980 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:26.145797 21980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688b466c3a6140a8938d5462eea364da, State: Initialized, Role: FOLLOWER
I20260620 19:21:26.145906 21980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260620 19:21:26.145972 21980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:26.145991 21980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:26.146015 21980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:26.146416 21980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260620 19:21:26.146485 21980 leader_election.cc:302] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688b466c3a6140a8938d5462eea364da; no voters:
I20260620 19:21:26.146588 21980 leader_election.cc:288] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:26.146638 21983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:26.146785 21980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:26.146814 21983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [term 1 LEADER]: Becoming Leader. State: Replica: 688b466c3a6140a8938d5462eea364da, State: Running, Role: LEADER
I20260620 19:21:26.147032 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 21912
I20260620 19:21:26.147043 21983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } }
I20260620 19:21:26.147130 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:26.147358 1031 external_mini_cluster.cc:1428] Setting key 6aae41eb4443ebec89f5d1c8c14e9a7e
I20260620 19:21:26.147671 21985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688b466c3a6140a8938d5462eea364da. Latest consensus state: current_term: 1 leader_uuid: "688b466c3a6140a8938d5462eea364da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } } }
I20260620 19:21:26.147743 21985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:26.147900 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688b466c3a6140a8938d5462eea364da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688b466c3a6140a8938d5462eea364da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39203 } } }
I20260620 19:21:26.148013 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da [sys.catalog]: This master's current role is: LEADER
W20260620 19:21:26.148762 22000 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:26.148855 22000 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:26.148918 21994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:26.149098 21994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:26.150458 21994 catalog_manager.cc:1347] Generated new cluster ID: e35dee9002c843a29cd3aa93eda06467
I20260620 19:21:26.150506 21994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:26 dist-test-slave-mwkb krb5kdc[21899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:21:26.158099 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:26.148047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:55981 (local address 127.0.0.1:39203)
0620 19:21:26.148446 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:26.148454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:26.148483 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:26.148585 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:26.148590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:26.148638 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:26.148769 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:26.149483 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:26.150741 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:26.151681 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:26.152226 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:26.152324 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:26.154926 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:26.154935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:26.154936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:26.154947 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:26.156597 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:26.157000 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:26.157002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:26.157002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:26.157116 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:26.157325 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:26.157326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:26.157326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:26.157566 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:26.157682 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:26.157811 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:21:26.157893 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":56,"threads_started":1}
I20260620 19:21:26.159054 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:26.166989 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:26.160476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:33107 (local address 127.0.0.1:39203)
0620 19:21:26.160505 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:26.160506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:26.160749 (+ 243us) server_negotiation.cc:407] Connection header received
0620 19:21:26.160829 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:26.160830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:26.160837 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:26.160858 (+ 21us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:26.161492 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:26.162216 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:26.163085 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:26.163377 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:26.163408 (+ 31us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:26.164733 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:26.164734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:26.164734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:26.164739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:26.165996 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:26.166317 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:26.166318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:26.166318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:26.166356 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:26.166527 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:26.166528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:26.166528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:26.166615 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:26.166679 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:26.166860 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 19:21:26.166910 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:21:26.167814 21994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:26.168200 21994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:26.172801 21994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688b466c3a6140a8938d5462eea364da: Generated new TSK 0
I20260620 19:21:26.172962 21994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:26.175124 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 21912
W20260620 19:21:26.179518 21897 connection.cc:535] client connection to 127.0.0.1:39203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39203): Connection reset by peer (error 104)
2026-06-20T19:21:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:26 dist-test-slave-mwkb krb5kdc[22011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:26 dist-test-slave-mwkb krb5kdc[22011](info): set up 2 sockets
Jun 20 19:21:26 dist-test-slave-mwkb krb5kdc[22011](info): commencing operation
krb5kdc: starting...
W20260620 19:21:27.176954 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:27 dist-test-slave-mwkb krb5kdc[22011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:27.281258 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:27.352284 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:27.352523 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:27.352558 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:27.353560 22024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:27.353641 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:27.353681 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:27.353711 22024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:27.353765 22024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:27.355165 22024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:27.355408 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:27.355630 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:27.360168 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:27.360195 22024 server_base.cc:1034] running on GCE node
W20260620 19:21:27.360348 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:27.360232 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:27.360620 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:27.360833 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:27.361994 22024 hybrid_clock.cc:648] HybridClock initialized: now 1781983287361977 us; error 26 us; skew 500 ppm
Jun 20 19:21:27 dist-test-slave-mwkb krb5kdc[22011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:27.364295 22024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:27.364698 22024 webserver.cc:460] Webserver started at http://127.0.0.1:45459/ using document root <none> and password file <none>
I20260620 19:21:27.364912 22024 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:27.364952 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:27.365119 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:27.366029 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "b293d78f0f954f63a5ac25f430697406"
format_stamp: "Formatted at 2026-06-20 19:21:27 on dist-test-slave-mwkb"
server_key: "a5f36a15b5cae1a4f7d33fc960530408"
server_key_iv: "0b1ed93d7cb02ff18e1816f21803b93f"
server_key_version: "encryptionkey@0"
I20260620 19:21:27.366348 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "b293d78f0f954f63a5ac25f430697406"
format_stamp: "Formatted at 2026-06-20 19:21:27 on dist-test-slave-mwkb"
server_key: "a5f36a15b5cae1a4f7d33fc960530408"
server_key_iv: "0b1ed93d7cb02ff18e1816f21803b93f"
server_key_version: "encryptionkey@0"
I20260620 19:21:27.367532 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:21:27.368131 22039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:27.368296 22024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:27.368373 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "b293d78f0f954f63a5ac25f430697406"
format_stamp: "Formatted at 2026-06-20 19:21:27 on dist-test-slave-mwkb"
server_key: "a5f36a15b5cae1a4f7d33fc960530408"
server_key_iv: "0b1ed93d7cb02ff18e1816f21803b93f"
server_key_version: "encryptionkey@0"
I20260620 19:21:27.368419 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:27.375783 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:27.377297 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:27.377386 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:27.380647 22024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37739
I20260620 19:21:27.380733 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37739 every 8 connection(s)
I20260620 19:21:27.380977 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:27.381757 22092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:27.383507 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406: Bootstrap starting.
I20260620 19:21:27.384018 22092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:27.384294 22092 log.cc:824] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:27.384604 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22024
I20260620 19:21:27.384720 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:27.384946 1031 external_mini_cluster.cc:1428] Setting key 8fd9403f9fe0cb8eddf915e34a792e22
I20260620 19:21:27.385161 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406: No bootstrap required, opened a new log
I20260620 19:21:27.387974 22092 raft_consensus.cc:348] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 19:21:27.388103 22092 raft_consensus.cc:374] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:27.388124 22092 raft_consensus.cc:729] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b293d78f0f954f63a5ac25f430697406, State: Initialized, Role: FOLLOWER
I20260620 19:21:27.388268 22092 consensus_queue.cc:260] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 19:21:27.388350 22092 raft_consensus.cc:388] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:27.388372 22092 raft_consensus.cc:482] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:27.388441 22092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:27.388960 22092 raft_consensus.cc:504] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 19:21:27.389070 22092 leader_election.cc:302] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b293d78f0f954f63a5ac25f430697406; no voters:
I20260620 19:21:27.389293 22092 leader_election.cc:288] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:27.389318 22097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:27.389431 22097 raft_consensus.cc:686] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [term 1 LEADER]: Becoming Leader. State: Replica: b293d78f0f954f63a5ac25f430697406, State: Running, Role: LEADER
I20260620 19:21:27.389554 22092 sys_catalog.cc:564] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:27.389572 22097 consensus_queue.cc:237] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 19:21:27.390287 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b293d78f0f954f63a5ac25f430697406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } } }
I20260620 19:21:27.390363 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:27.390499 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b293d78f0f954f63a5ac25f430697406. Latest consensus state: current_term: 1 leader_uuid: "b293d78f0f954f63a5ac25f430697406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b293d78f0f954f63a5ac25f430697406" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } } }
I20260620 19:21:27.390558 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:27.390581 22107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:27.391058 22107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:27.391124 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:27.385410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:50237 (local address 127.0.0.1:37739)
0620 19:21:27.385586 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:27.385595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:27.385627 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:21:27.386793 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:27.386800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:27.386858 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:27.386993 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:27.387206 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:27.388057 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:27.388972 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:27.389578 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:27.389979 (+ 401us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:27.390402 (+ 423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:27.390405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:27.390406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:27.390619 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:27.390655 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:27.390668 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:27.390791 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:27.390916 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:21:27.390940 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260620 19:21:27.392004 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:27.392650 22107 catalog_manager.cc:1347] Generated new cluster ID: b03e69595a6643ebb8ba052368d0892d
I20260620 19:21:27.392694 22107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:27.404148 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:27.393530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:57389 (local address 127.0.0.1:37739)
0620 19:21:27.393568 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:27.393569 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:27.401265 (+ 7696us) server_negotiation.cc:407] Connection header received
0620 19:21:27.401352 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:27.401353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:27.401362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:27.401433 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:27.401598 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:27.402053 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:27.402964 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:27.403177 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:27.403263 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:27.403891 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:27.403893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:27.403893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:27.403897 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:27.403910 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:27.403914 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:27.403971 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:27.404047 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:21:27.404056 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:27.414320 22107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:27.414808 22107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:27.418659 22107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b293d78f0f954f63a5ac25f430697406: Generated new TSK 0
I20260620 19:21:27.418812 22107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:27.425731 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22024
W20260620 19:21:27.429703 22009 connection.cc:535] client connection to 127.0.0.1:37739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37739): Connection reset by peer (error 104)
2026-06-20T19:21:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:27 dist-test-slave-mwkb krb5kdc[22124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:27 dist-test-slave-mwkb krb5kdc[22124](info): set up 2 sockets
Jun 20 19:21:27 dist-test-slave-mwkb krb5kdc[22124](info): commencing operation
krb5kdc: starting...
W20260620 19:21:28.421047 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:28 dist-test-slave-mwkb krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:28.528563 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:28.604144 22137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:28.604331 22137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:28.604353 22137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:28.605429 22137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:28.605463 22137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:28.605477 22137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:28.605490 22137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:28.605501 22137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:28.606863 22137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:28.607034 22137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:28.607182 22137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:21:28.611671 22137 server_base.cc:1034] running on GCE node
W20260620 19:21:28.611657 22143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:28.611657 22142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:28.611650 22145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:28.612087 22137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:28.612351 22137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:28.613497 22137 hybrid_clock.cc:648] HybridClock initialized: now 1781983288613480 us; error 37 us; skew 500 ppm
Jun 20 19:21:28 dist-test-slave-mwkb krb5kdc[22124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:28.615772 22137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:28.616151 22137 webserver.cc:460] Webserver started at http://127.0.0.1:45381/ using document root <none> and password file <none>
I20260620 19:21:28.616369 22137 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:28.616430 22137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:28.616566 22137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:28.617516 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "a218f184406e4433a2ce760707f19fab"
format_stamp: "Formatted at 2026-06-20 19:21:28 on dist-test-slave-mwkb"
server_key: "78e2adf3b6efd65c204b3b2dbb124c9f"
server_key_iv: "26c972a0c2d76c3f380da07b67941e68"
server_key_version: "encryptionkey@0"
I20260620 19:21:28.617813 22137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "a218f184406e4433a2ce760707f19fab"
format_stamp: "Formatted at 2026-06-20 19:21:28 on dist-test-slave-mwkb"
server_key: "78e2adf3b6efd65c204b3b2dbb124c9f"
server_key_iv: "26c972a0c2d76c3f380da07b67941e68"
server_key_version: "encryptionkey@0"
I20260620 19:21:28.618908 22137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:28.619558 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:28.619721 22137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:28.619820 22137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "a218f184406e4433a2ce760707f19fab"
format_stamp: "Formatted at 2026-06-20 19:21:28 on dist-test-slave-mwkb"
server_key: "78e2adf3b6efd65c204b3b2dbb124c9f"
server_key_iv: "26c972a0c2d76c3f380da07b67941e68"
server_key_version: "encryptionkey@0"
I20260620 19:21:28.619895 22137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:28.622809 22137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:28.624477 22137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:28.624624 22137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:28.628036 22137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45909
I20260620 19:21:28.628082 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45909 every 8 connection(s)
I20260620 19:21:28.628389 22137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:28.629411 22205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:28.632423 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab: Bootstrap starting.
I20260620 19:21:28.633113 22205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:28.633430 22205 log.cc:824] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:28.633772 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22137
I20260620 19:21:28.633881 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:28.634059 1031 external_mini_cluster.cc:1428] Setting key 52c887d99cc5fc760a611107913866b5
I20260620 19:21:28.636932 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:28.634623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:41981 (local address 127.0.0.1:45909)
0620 19:21:28.635025 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:28.635046 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:28.635075 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:28.635566 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:28.635574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:28.635607 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:28.635983 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:28.636396 (+ 413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:28.636399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:28.636399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:28.636403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:28.636435 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:28.636447 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:28.636702 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:28.636730 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 19:21:28.636766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":81,"threads_started":1}
I20260620 19:21:28.637735 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:28.639670 22205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab: No bootstrap required, opened a new log
I20260620 19:21:28.641028 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:28.639907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:53035 (local address 127.0.0.1:45909)
0620 19:21:28.639942 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:28.639944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:28.640088 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:21:28.640166 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:28.640167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:28.640171 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:28.640218 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:28.640795 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:28.640796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:28.640796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:28.640801 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:28.640813 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:28.640817 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:28.640877 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:28.640952 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:21:28.640975 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:21:28.642155 22205 raft_consensus.cc:348] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260620 19:21:28.642298 22205 raft_consensus.cc:374] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:28.642342 22205 raft_consensus.cc:729] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a218f184406e4433a2ce760707f19fab, State: Initialized, Role: FOLLOWER
I20260620 19:21:28.642458 22205 consensus_queue.cc:260] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260620 19:21:28.642547 22205 raft_consensus.cc:388] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:28.642588 22205 raft_consensus.cc:482] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:28.642637 22205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:28.643123 22205 raft_consensus.cc:504] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260620 19:21:28.643260 22205 leader_election.cc:302] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a218f184406e4433a2ce760707f19fab; no voters:
I20260620 19:21:28.643424 22205 leader_election.cc:288] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:28.643450 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:28.643617 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [term 1 LEADER]: Becoming Leader. State: Replica: a218f184406e4433a2ce760707f19fab, State: Running, Role: LEADER
I20260620 19:21:28.643657 22205 sys_catalog.cc:564] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:28.643743 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } }
I20260620 19:21:28.644160 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [sys.catalog]: SysCatalogTable state changed. Reason: New leader a218f184406e4433a2ce760707f19fab. Latest consensus state: current_term: 1 leader_uuid: "a218f184406e4433a2ce760707f19fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260620 19:21:28.644166 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a218f184406e4433a2ce760707f19fab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a218f184406e4433a2ce760707f19fab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45909 } } }
I20260620 19:21:28.644325 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:28.644315 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:28.644547 22224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:28.645176 22224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:28.646528 22224 catalog_manager.cc:1347] Generated new cluster ID: b379f48fda1c4e3da1b9bc5c23a1d50f
I20260620 19:21:28.646574 22224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:28.650270 22224 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:28.650399 22224 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:28.654870 22224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a218f184406e4433a2ce760707f19fab: Generated new TSK 0
I20260620 19:21:28.655040 22224 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:28.658665 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22137
2026-06-20T19:21:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1232 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:28 dist-test-slave-mwkb krb5kdc[22236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:28 dist-test-slave-mwkb krb5kdc[22236](info): set up 2 sockets
Jun 20 19:21:28 dist-test-slave-mwkb krb5kdc[22236](info): commencing operation
krb5kdc: starting...
W20260620 19:21:29.680927 1031 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:29.787744 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:29.862164 22249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:29.862344 22249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:29.862367 22249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:29.863482 22249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:29.863523 22249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:29.863538 22249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:29.863549 22249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:29.863562 22249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:29.864964 22249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:29.865214 22249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:29.865432 22249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:29.870085 22254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:29.870153 22255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:29.870306 22249 server_base.cc:1034] running on GCE node
W20260620 19:21:29.870072 22257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:29.870661 22249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:29.870961 22249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:29.872117 22249 hybrid_clock.cc:648] HybridClock initialized: now 1781983289872107 us; error 37 us; skew 500 ppm
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:29.874347 22249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:29.874744 22249 webserver.cc:460] Webserver started at http://127.0.0.1:33997/ using document root <none> and password file <none>
I20260620 19:21:29.874956 22249 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:29.875005 22249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:29.875144 22249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:29.876091 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "d8ffb69baf1c468d8e608f1cf8ebb857"
format_stamp: "Formatted at 2026-06-20 19:21:29 on dist-test-slave-mwkb"
server_key: "d49b736cd27ed16475b8248ae349380f"
server_key_iv: "4b70fdf48082e0e59efca6a666e3a69b"
server_key_version: "encryptionkey@0"
I20260620 19:21:29.876420 22249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "d8ffb69baf1c468d8e608f1cf8ebb857"
format_stamp: "Formatted at 2026-06-20 19:21:29 on dist-test-slave-mwkb"
server_key: "d49b736cd27ed16475b8248ae349380f"
server_key_iv: "4b70fdf48082e0e59efca6a666e3a69b"
server_key_version: "encryptionkey@0"
I20260620 19:21:29.877786 22249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:29.878476 22264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:29.878664 22249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:29.878748 22249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d8ffb69baf1c468d8e608f1cf8ebb857"
format_stamp: "Formatted at 2026-06-20 19:21:29 on dist-test-slave-mwkb"
server_key: "d49b736cd27ed16475b8248ae349380f"
server_key_iv: "4b70fdf48082e0e59efca6a666e3a69b"
server_key_version: "encryptionkey@0"
I20260620 19:21:29.878821 22249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:29.887076 22249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:29.888772 22249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:29.888921 22249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:29.894833 22249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42307
I20260620 19:21:29.894862 22316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42307 every 8 connection(s)
I20260620 19:21:29.895202 22249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:29.896353 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:29.898514 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857: Bootstrap starting.
I20260620 19:21:29.899016 22317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:29.899242 22317 log.cc:824] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:29.900134 22317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857: No bootstrap required, opened a new log
I20260620 19:21:29.901533 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22249
I20260620 19:21:29.901646 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:29.901705 22317 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260620 19:21:29.901860 22317 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:29.901870 1031 external_mini_cluster.cc:1428] Setting key feb15946f854fb4e5f920ea0c9631225
I20260620 19:21:29.901885 22317 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ffb69baf1c468d8e608f1cf8ebb857, State: Initialized, Role: FOLLOWER
I20260620 19:21:29.902040 22317 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260620 19:21:29.902127 22317 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:29.902169 22317 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:29.902218 22317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:29.903091 22317 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260620 19:21:29.903213 22317 leader_election.cc:302] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8ffb69baf1c468d8e608f1cf8ebb857; no voters:
I20260620 19:21:29.903365 22317 leader_election.cc:288] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:29.903398 22322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:29.903548 22322 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [term 1 LEADER]: Becoming Leader. State: Replica: d8ffb69baf1c468d8e608f1cf8ebb857, State: Running, Role: LEADER
I20260620 19:21:29.903630 22317 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:29.903638 22322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } }
I20260620 19:21:29.904008 22322 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8ffb69baf1c468d8e608f1cf8ebb857. Latest consensus state: current_term: 1 leader_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } } }
I20260620 19:21:29.904023 22323 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8ffb69baf1c468d8e608f1cf8ebb857" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42307 } } }
I20260620 19:21:29.904096 22322 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:29.904102 22323 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:29.904312 22329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:29.904767 22329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:29.906433 22329 catalog_manager.cc:1347] Generated new cluster ID: 5cd465980ce94eeeaa75ed3d1ee186bf
I20260620 19:21:29.906484 22329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:21:29.913272 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:29.902400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:35771 (local address 127.0.0.1:42307)
0620 19:21:29.902589 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:29.902604 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:29.902627 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:21:29.902717 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:29.902722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:29.902762 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:29.902892 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:29.903582 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:29.905025 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:29.906268 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:29.906877 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:29.906973 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:29.909575 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:29.909579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:29.909579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:29.909590 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:29.911423 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:29.911878 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:29.911882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:29.911882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:29.912034 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:29.912270 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:29.912271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:29.912271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:29.912595 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:29.912764 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:29.912844 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:21:29.913055 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260620 19:21:29.914188 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:29.921120 22329 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:29.921782 22329 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:29.922401 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:29.915462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:36341 (local address 127.0.0.1:42307)
0620 19:21:29.915503 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:29.915505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:29.915649 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:21:29.915716 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:29.915716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:29.915725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:29.915772 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:29.916338 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:29.917025 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:29.918040 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:29.918342 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:29.918398 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:29.919701 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:29.919702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:29.919702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:29.919709 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:29.921296 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:29.921645 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:29.921646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:29.921647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:29.921698 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:29.921933 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:29.921934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:29.921934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:29.922036 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:29.922139 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:29.922259 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:21:29.922317 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:21:29.928325 22329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8ffb69baf1c468d8e608f1cf8ebb857: Generated new TSK 0
I20260620 19:21:29.928481 22329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:29.931267 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22249
W20260620 19:21:29.935519 22234 connection.cc:535] client connection to 127.0.0.1:42307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42307): Connection reset by peer (error 104)
2026-06-20T19:21:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22348](info): set up 2 sockets
Jun 20 19:21:29 dist-test-slave-mwkb krb5kdc[22348](info): commencing operation
krb5kdc: starting...
W20260620 19:21:30.928799 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:31 dist-test-slave-mwkb krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:31.040235 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:31.119948 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:31.120170 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:31.120206 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:31.121409 22361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:31.121469 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:31.121495 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:31.121615 22361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:31.121712 22361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:31.123227 22361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:31.123477 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:31.123708 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:31.128434 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:31.128458 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:31.128598 22361 server_base.cc:1034] running on GCE node
W20260620 19:21:31.128635 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:31.128857 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:31.129163 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:31.130339 22361 hybrid_clock.cc:648] HybridClock initialized: now 1781983291130318 us; error 31 us; skew 500 ppm
Jun 20 19:21:31 dist-test-slave-mwkb krb5kdc[22348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:31.135790 22361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:31.136222 22361 webserver.cc:460] Webserver started at http://127.0.0.1:41191/ using document root <none> and password file <none>
I20260620 19:21:31.136430 22361 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:31.136502 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:31.136643 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:31.137650 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6"
format_stamp: "Formatted at 2026-06-20 19:21:31 on dist-test-slave-mwkb"
server_key: "6506cb09d073d3776d2fad665762affd"
server_key_iv: "2beab40cd2760189895a911e6cd8455e"
server_key_version: "encryptionkey@0"
I20260620 19:21:31.137985 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6"
format_stamp: "Formatted at 2026-06-20 19:21:31 on dist-test-slave-mwkb"
server_key: "6506cb09d073d3776d2fad665762affd"
server_key_iv: "2beab40cd2760189895a911e6cd8455e"
server_key_version: "encryptionkey@0"
I20260620 19:21:31.139173 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 19:21:31.139894 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:31.140058 22361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:31.140163 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6"
format_stamp: "Formatted at 2026-06-20 19:21:31 on dist-test-slave-mwkb"
server_key: "6506cb09d073d3776d2fad665762affd"
server_key_iv: "2beab40cd2760189895a911e6cd8455e"
server_key_version: "encryptionkey@0"
I20260620 19:21:31.140236 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:31.153048 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:31.155172 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:31.155328 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:31.159403 22361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34709
I20260620 19:21:31.159432 22428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34709 every 8 connection(s)
I20260620 19:21:31.159799 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:31.160575 22429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:31.162724 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6: Bootstrap starting.
I20260620 19:21:31.163267 22429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:31.163537 22429 log.cc:824] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:31.164289 22429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6: No bootstrap required, opened a new log
I20260620 19:21:31.164270 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22361
I20260620 19:21:31.164384 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:31.164597 1031 external_mini_cluster.cc:1428] Setting key 4f2ce123fa59f95d4705874c7d4885d7
I20260620 19:21:31.166002 22429 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } }
I20260620 19:21:31.166139 22429 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:31.166193 22429 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd491bf1f8684ba9bf61e7ffdb5982e6, State: Initialized, Role: FOLLOWER
I20260620 19:21:31.166379 22429 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } }
I20260620 19:21:31.166456 22429 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:31.166493 22429 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:31.166535 22429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:31.167167 22429 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } }
I20260620 19:21:31.167292 22429 leader_election.cc:302] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd491bf1f8684ba9bf61e7ffdb5982e6; no voters:
I20260620 19:21:31.167424 22429 leader_election.cc:288] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:31.167469 22434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:31.167632 22429 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:31.167639 22434 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [term 1 LEADER]: Becoming Leader. State: Replica: fd491bf1f8684ba9bf61e7ffdb5982e6, State: Running, Role: LEADER
I20260620 19:21:31.167783 22434 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } }
I20260620 19:21:31.168097 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } } }
I20260620 19:21:31.168206 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:31.168416 22443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:31.168919 22436 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd491bf1f8684ba9bf61e7ffdb5982e6. Latest consensus state: current_term: 1 leader_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd491bf1f8684ba9bf61e7ffdb5982e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34709 } } }
I20260620 19:21:31.169028 22436 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:31.169054 22443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:31.170699 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:31.165052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:34037 (local address 127.0.0.1:34709)
0620 19:21:31.165287 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:31.165295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:31.165322 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:21:31.165991 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:31.166004 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:31.166044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:31.166162 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:31.166376 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:31.167849 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:31.168854 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:31.169334 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:31.169448 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:31.170012 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:31.170016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:31.170016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:31.170188 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:31.170223 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:31.170233 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:31.170307 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:31.170443 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:21:31.170486 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
I20260620 19:21:31.170987 22443 catalog_manager.cc:1347] Generated new cluster ID: e9a61c84b1b84fa58a75ae41e9957fee
I20260620 19:21:31.171041 22443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:31.171424 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:31.175623 22433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:31.172606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:43639 (local address 127.0.0.1:34709)
0620 19:21:31.172646 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:31.172648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:31.173211 (+ 563us) server_negotiation.cc:407] Connection header received
0620 19:21:31.173271 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:31.173272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:31.173281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:31.173328 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:31.173464 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:31.173837 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:31.174548 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:31.174769 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:31.174848 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:31.175410 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:31.175411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:31.175411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:31.175414 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:31.175424 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:31.175427 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:31.175468 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:31.175517 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:21:31.175527 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:21:31.178129 22443 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:31.178723 22443 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:31.181439 22443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd491bf1f8684ba9bf61e7ffdb5982e6: Generated new TSK 0
I20260620 19:21:31.181577 22443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:31.182996 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22361
W20260620 19:21:31.187602 22346 connection.cc:535] client connection to 127.0.0.1:34709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34709): Connection reset by peer (error 104)
2026-06-20T19:21:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:31 dist-test-slave-mwkb krb5kdc[22460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:31 dist-test-slave-mwkb krb5kdc[22460](info): set up 2 sockets
Jun 20 19:21:31 dist-test-slave-mwkb krb5kdc[22460](info): commencing operation
krb5kdc: starting...
W20260620 19:21:32.169232 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:32 dist-test-slave-mwkb krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:21:32.277045 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:32.353633 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:32.353850 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:32.353886 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:32.355014 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:32.355082 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:32.355108 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:32.355139 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:32.355161 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:32.356515 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:32.356784 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:32.357007 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:32.362357 22481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:32.362425 22473 server_base.cc:1034] running on GCE node
W20260620 19:21:32.362382 22478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:32.362382 22479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:32.362871 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:32.363145 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:32.364290 22473 hybrid_clock.cc:648] HybridClock initialized: now 1781983292364273 us; error 28 us; skew 500 ppm
Jun 20 19:21:32 dist-test-slave-mwkb krb5kdc[22460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:32.366600 22473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:21:32.366989 22473 webserver.cc:460] Webserver started at http://127.0.0.1:34333/ using document root <none> and password file <none>
I20260620 19:21:32.367199 22473 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:32.367264 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:32.367401 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:32.368345 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "b22153e51528400bb753c28aebc5ba6f"
format_stamp: "Formatted at 2026-06-20 19:21:32 on dist-test-slave-mwkb"
server_key: "7ae81589eadb70b2a6eded1869c29f78"
server_key_iv: "421716769fa7daccfa63a4aba95b3370"
server_key_version: "encryptionkey@0"
I20260620 19:21:32.368688 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "b22153e51528400bb753c28aebc5ba6f"
format_stamp: "Formatted at 2026-06-20 19:21:32 on dist-test-slave-mwkb"
server_key: "7ae81589eadb70b2a6eded1869c29f78"
server_key_iv: "421716769fa7daccfa63a4aba95b3370"
server_key_version: "encryptionkey@0"
I20260620 19:21:32.370018 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:21:32.370725 22488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:32.370891 22473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:32.370985 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "b22153e51528400bb753c28aebc5ba6f"
format_stamp: "Formatted at 2026-06-20 19:21:32 on dist-test-slave-mwkb"
server_key: "7ae81589eadb70b2a6eded1869c29f78"
server_key_iv: "421716769fa7daccfa63a4aba95b3370"
server_key_version: "encryptionkey@0"
I20260620 19:21:32.371073 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:32.374140 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:32.375789 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:32.375962 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:32.379678 22473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38179
I20260620 19:21:32.379704 22540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38179 every 8 connection(s)
I20260620 19:21:32.379998 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:32.380789 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22473
I20260620 19:21:32.380903 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:32.380937 22541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:32.381124 1031 external_mini_cluster.cc:1428] Setting key 50c23fa3c0f15a988cc7c73243e8b552
I20260620 19:21:32.383275 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:32.381556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:45603 (local address 127.0.0.1:38179)
0620 19:21:32.381702 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:32.381709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:32.381736 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:21:32.382183 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:32.382191 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:32.382221 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:32.382352 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:32.382885 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:32.382888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:32.382889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:32.382893 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:32.382932 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:32.382946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:32.383025 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:32.383061 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 19:21:32.383097 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260620 19:21:32.385041 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f: Bootstrap starting.
I20260620 19:21:32.386513 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:32.387506 22541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:32.388207 22541 log.cc:824] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:32.390276 22541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f: No bootstrap required, opened a new log
I20260620 19:21:32.391289 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:32.389808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:59053 (local address 127.0.0.1:38179)
0620 19:21:32.389904 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:32.389907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:32.390172 (+ 265us) server_negotiation.cc:407] Connection header received
0620 19:21:32.390205 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:32.390206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:32.390212 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:32.390256 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:32.391026 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:32.391027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:32.391027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:32.391031 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:32.391042 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:32.391045 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:32.391097 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:32.391165 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:21:32.391176 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:32.393294 22541 raft_consensus.cc:348] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260620 19:21:32.393748 22541 raft_consensus.cc:374] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:32.393839 22541 raft_consensus.cc:729] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b22153e51528400bb753c28aebc5ba6f, State: Initialized, Role: FOLLOWER
I20260620 19:21:32.393987 22541 consensus_queue.cc:260] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260620 19:21:32.394121 22541 raft_consensus.cc:388] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:32.394161 22541 raft_consensus.cc:482] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:32.394217 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:32.395270 22541 raft_consensus.cc:504] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260620 19:21:32.395424 22541 leader_election.cc:302] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b22153e51528400bb753c28aebc5ba6f; no voters:
I20260620 19:21:32.396008 22541 leader_election.cc:288] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:32.396236 22551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:32.396340 22541 sys_catalog.cc:564] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:32.397011 22551 raft_consensus.cc:686] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [term 1 LEADER]: Becoming Leader. State: Replica: b22153e51528400bb753c28aebc5ba6f, State: Running, Role: LEADER
I20260620 19:21:32.397379 22551 consensus_queue.cc:237] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260620 19:21:32.398036 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b22153e51528400bb753c28aebc5ba6f. Latest consensus state: current_term: 1 leader_uuid: "b22153e51528400bb753c28aebc5ba6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } } }
I20260620 19:21:32.398147 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:32.398751 22552 sys_catalog.cc:455] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b22153e51528400bb753c28aebc5ba6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b22153e51528400bb753c28aebc5ba6f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } } }
I20260620 19:21:32.398869 22552 sys_catalog.cc:458] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:32.398893 22562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:32.399349 22562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:32.400815 22562 catalog_manager.cc:1347] Generated new cluster ID: 0e91927b6f954e76a734840da65b1caf
I20260620 19:21:32.400887 22562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:32.416761 22562 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:32.416901 22562 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:32.421775 22562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b22153e51528400bb753c28aebc5ba6f: Generated new TSK 0
I20260620 19:21:32.421908 22562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:32.426272 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22473
2026-06-20T19:21:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:32 dist-test-slave-mwkb krb5kdc[22572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:32 dist-test-slave-mwkb krb5kdc[22572](info): set up 2 sockets
Jun 20 19:21:32 dist-test-slave-mwkb krb5kdc[22572](info): commencing operation
krb5kdc: starting...
W20260620 19:21:33.420955 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:33.527460 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:40185
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:40185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:33.602406 22585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:33.602581 22585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:33.602602 22585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:33.603729 22585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:33.603768 22585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:33.603782 22585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:33.603832 22585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:33.603850 22585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:33.605283 22585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:40185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:40185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:33.605521 22585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:33.605727 22585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:33.610227 22593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:33.610273 22590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:33.610239 22591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:33.610455 22585 server_base.cc:1034] running on GCE node
I20260620 19:21:33.610671 22585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:33.610939 22585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:33.612077 22585 hybrid_clock.cc:648] HybridClock initialized: now 1781983293612066 us; error 26 us; skew 500 ppm
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:33.614284 22585 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:33.614665 22585 webserver.cc:460] Webserver started at http://127.1.1.254:35799/ using document root <none> and password file <none>
I20260620 19:21:33.614884 22585 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:33.614928 22585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:33.615073 22585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:33.615996 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "d82e127b59d944f3a5143871eebd36aa"
format_stamp: "Formatted at 2026-06-20 19:21:33 on dist-test-slave-mwkb"
server_key: "09212647664fa196f7ae9b67f9415bbe"
server_key_iv: "44883b207de025ecebec51a6459a8d80"
server_key_version: "encryptionkey@0"
I20260620 19:21:33.616369 22585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "d82e127b59d944f3a5143871eebd36aa"
format_stamp: "Formatted at 2026-06-20 19:21:33 on dist-test-slave-mwkb"
server_key: "09212647664fa196f7ae9b67f9415bbe"
server_key_iv: "44883b207de025ecebec51a6459a8d80"
server_key_version: "encryptionkey@0"
I20260620 19:21:33.617947 22585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:21:33.618633 22600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:33.618860 22585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:33.618950 22585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d82e127b59d944f3a5143871eebd36aa"
format_stamp: "Formatted at 2026-06-20 19:21:33 on dist-test-slave-mwkb"
server_key: "09212647664fa196f7ae9b67f9415bbe"
server_key_iv: "44883b207de025ecebec51a6459a8d80"
server_key_version: "encryptionkey@0"
I20260620 19:21:33.619030 22585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:33.628998 22585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:33.630542 22585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:33.630676 22585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:33.633905 22585 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:40185
I20260620 19:21:33.633944 22652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:40185 every 8 connection(s)
I20260620 19:21:33.634246 22585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:33.635048 22653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:33.636860 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa: Bootstrap starting.
I20260620 19:21:33.637403 22653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:33.637668 22653 log.cc:824] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:33.638236 22653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa: No bootstrap required, opened a new log
I20260620 19:21:33.639537 22653 raft_consensus.cc:348] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } }
I20260620 19:21:33.639662 22653 raft_consensus.cc:374] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:33.639717 22653 raft_consensus.cc:729] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d82e127b59d944f3a5143871eebd36aa, State: Initialized, Role: FOLLOWER
I20260620 19:21:33.639894 22653 consensus_queue.cc:260] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } }
I20260620 19:21:33.639969 22653 raft_consensus.cc:388] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:33.640017 22653 raft_consensus.cc:482] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:33.640067 22653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:33.640856 22653 raft_consensus.cc:504] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } }
I20260620 19:21:33.640971 22653 leader_election.cc:302] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d82e127b59d944f3a5143871eebd36aa; no voters:
I20260620 19:21:33.641147 22653 leader_election.cc:288] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:33.641151 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22585
I20260620 19:21:33.641175 22656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:33.641240 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:33.641290 22656 raft_consensus.cc:686] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [term 1 LEADER]: Becoming Leader. State: Replica: d82e127b59d944f3a5143871eebd36aa, State: Running, Role: LEADER
I20260620 19:21:33.641345 22653 sys_catalog.cc:564] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:33.641360 22656 consensus_queue.cc:237] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } }
I20260620 19:21:33.641415 1031 external_mini_cluster.cc:1428] Setting key 230b0c6d4c658bbcdd84b14dd36b7194
I20260620 19:21:33.641654 22658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d82e127b59d944f3a5143871eebd36aa. Latest consensus state: current_term: 1 leader_uuid: "d82e127b59d944f3a5143871eebd36aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } } }
I20260620 19:21:33.641736 22658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:33.641933 22665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:33.642838 22665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:33.642360 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d82e127b59d944f3a5143871eebd36aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d82e127b59d944f3a5143871eebd36aa" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 40185 } } }
I20260620 19:21:33.643209 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:33.644156 22665 catalog_manager.cc:1347] Generated new cluster ID: ae6271f5af4c430cb4945aa7700386f1
I20260620 19:21:33.644200 22665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:33.651588 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:33.642008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35875 (local address 127.1.1.254:40185)
0620 19:21:33.642152 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:33.642159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:33.642193 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:21:33.642333 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:33.642340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:33.642389 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:33.642518 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:33.643159 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:33.644325 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:33.645379 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:33.645998 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:33.646133 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:33.648436 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:33.648439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:33.648439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:33.648449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:33.650085 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:33.650491 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:33.650494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:33.650494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:33.650591 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:33.650816 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:33.650817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:33.650817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:33.651066 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:33.651194 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:33.651332 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:21:33.651424 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260620 19:21:33.652565 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:33.660564 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:33.653765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57011 (local address 127.1.1.254:40185)
0620 19:21:33.653791 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:33.653793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:33.653888 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:21:33.653947 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:33.653948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:33.653956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:33.654007 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:33.654607 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:33.655309 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:33.656230 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:33.656475 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:33.656540 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:33.657880 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:33.657880 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:33.657881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:33.657887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:33.659339 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:33.659754 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:33.659755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:33.659755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:33.659808 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:33.660040 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:33.660041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:33.660041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:33.660154 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:33.660246 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:33.660413 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 19:21:33.660478 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:21:33.661006 22665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:33.661471 22665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:33.664733 22665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d82e127b59d944f3a5143871eebd36aa: Generated new TSK 0
I20260620 19:21:33.664882 22665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:33.666685 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22585
W20260620 19:21:33.670234 22570 connection.cc:535] client connection to 127.1.1.254:40185 recv error: Network error: failed to read from TLS socket (remote: 127.1.1.254:40185): Connection reset by peer (error 104)
2026-06-20T19:21:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22684](info): set up 2 sockets
Jun 20 19:21:33 dist-test-slave-mwkb krb5kdc[22684](info): commencing operation
krb5kdc: starting...
W20260620 19:21:34.648577 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:34 dist-test-slave-mwkb krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:34.755424 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:32797
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:32797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:34.829402 22697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:34.829567 22697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:34.829587 22697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:34.830637 22697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:34.830677 22697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:34.830689 22697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:34.830745 22697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:34.830789 22697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:34.832173 22697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:32797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:32797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:34.832391 22697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:34.832587 22697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:21:34.837224 22697 server_base.cc:1034] running on GCE node
W20260620 19:21:34.837234 22705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:34.837201 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:34.837324 22703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:34.837616 22697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:34.837888 22697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:34.839020 22697 hybrid_clock.cc:648] HybridClock initialized: now 1781983294839008 us; error 27 us; skew 500 ppm
Jun 20 19:21:34 dist-test-slave-mwkb krb5kdc[22684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983294, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:34.841279 22697 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:34.841656 22697 webserver.cc:460] Webserver started at http://127.1.1.254:34763/ using document root <none> and password file <none>
I20260620 19:21:34.841849 22697 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:34.841905 22697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:34.842031 22697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:34.842880 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "d5af6542ffb9451c8d9f0d375671acc0"
format_stamp: "Formatted at 2026-06-20 19:21:34 on dist-test-slave-mwkb"
server_key: "3e140a93bffaf85210f058417be04d6d"
server_key_iv: "7224f7618786adbb8600505177a622cb"
server_key_version: "encryptionkey@0"
I20260620 19:21:34.843185 22697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "d5af6542ffb9451c8d9f0d375671acc0"
format_stamp: "Formatted at 2026-06-20 19:21:34 on dist-test-slave-mwkb"
server_key: "3e140a93bffaf85210f058417be04d6d"
server_key_iv: "7224f7618786adbb8600505177a622cb"
server_key_version: "encryptionkey@0"
I20260620 19:21:34.844377 22697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:34.845350 22712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:34.845664 22697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:34.845772 22697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "d5af6542ffb9451c8d9f0d375671acc0"
format_stamp: "Formatted at 2026-06-20 19:21:34 on dist-test-slave-mwkb"
server_key: "3e140a93bffaf85210f058417be04d6d"
server_key_iv: "7224f7618786adbb8600505177a622cb"
server_key_version: "encryptionkey@0"
I20260620 19:21:34.845846 22697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:34.876212 22697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:34.879091 22697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:34.879246 22697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:34.883296 22697 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:32797
I20260620 19:21:34.883327 22764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:32797 every 8 connection(s)
I20260620 19:21:34.883667 22697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:34.884536 22765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:34.888108 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: Bootstrap starting.
I20260620 19:21:34.889377 22765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:34.889513 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22697
I20260620 19:21:34.889678 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:34.889925 1031 external_mini_cluster.cc:1428] Setting key 143e20b995d0d2783ada726b51ca6747
I20260620 19:21:34.889921 22765 log.cc:824] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:34.891292 22765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: No bootstrap required, opened a new log
I20260620 19:21:34.895331 22765 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } }
I20260620 19:21:34.895542 22765 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:34.895579 22765 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5af6542ffb9451c8d9f0d375671acc0, State: Initialized, Role: FOLLOWER
I20260620 19:21:34.895705 22765 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } }
I20260620 19:21:34.895792 22765 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:34.895830 22765 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:34.895875 22765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:34.896405 22765 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } }
I20260620 19:21:34.896528 22765 leader_election.cc:302] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5af6542ffb9451c8d9f0d375671acc0; no voters:
I20260620 19:21:34.896780 22765 leader_election.cc:288] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:34.896806 22770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:34.896950 22770 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [term 1 LEADER]: Becoming Leader. State: Replica: d5af6542ffb9451c8d9f0d375671acc0, State: Running, Role: LEADER
I20260620 19:21:34.897037 22770 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } }
I20260620 19:21:34.897118 22765 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:34.897390 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5af6542ffb9451c8d9f0d375671acc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } } }
I20260620 19:21:34.897459 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:34.897588 22770 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5af6542ffb9451c8d9f0d375671acc0. Latest consensus state: current_term: 1 leader_uuid: "d5af6542ffb9451c8d9f0d375671acc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5af6542ffb9451c8d9f0d375671acc0" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 32797 } } }
I20260620 19:21:34.897651 22770 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0 [sys.catalog]: This master's current role is: LEADER
W20260620 19:21:34.898922 22786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:34.898984 22786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:34.899114 22775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:34.899327 22775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:34.900087 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:34.890677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57853 (local address 127.1.1.254:32797)
0620 19:21:34.893486 (+ 2809us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:34.893516 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:34.893637 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:21:34.895254 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:34.895266 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:34.895348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:34.895535 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:34.895849 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:34.896820 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:34.897977 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:34.898458 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:34.898562 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:34.899282 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:34.899286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:34.899286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:34.899469 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:34.899519 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:34.899535 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:34.899610 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:34.899868 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 19:21:34.899893 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2632,"thread_start_us":120,"threads_started":1}
I20260620 19:21:34.900637 22775 catalog_manager.cc:1347] Generated new cluster ID: 2edb5e3420f0430f86011c3e8d8620b2
I20260620 19:21:34.900686 22775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:34.900995 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:34.905660 22775 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:34.905849 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:34.902416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58611 (local address 127.1.1.254:32797)
0620 19:21:34.902455 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:34.902456 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:34.902537 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:21:34.902580 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:34.902581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:34.902589 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:34.902659 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:34.902786 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:34.903468 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:34.904450 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:34.904807 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:34.904878 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:34.905586 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:34.905587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:34.905587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:34.905592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:34.905604 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:34.905608 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:34.905663 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:34.905779 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:21:34.905788 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:34.906306 22775 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:34.914594 22775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5af6542ffb9451c8d9f0d375671acc0: Generated new TSK 0
I20260620 19:21:34.914745 22775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:34.920061 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22697
W20260620 19:21:34.924093 22682 connection.cc:535] client connection to 127.1.1.254:32797 recv error: Network error: failed to read from TLS socket (remote: 127.1.1.254:32797): Connection reset by peer (error 104)
2026-06-20T19:21:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:34 dist-test-slave-mwkb krb5kdc[22796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:34 dist-test-slave-mwkb krb5kdc[22796](info): set up 2 sockets
Jun 20 19:21:34 dist-test-slave-mwkb krb5kdc[22796](info): commencing operation
krb5kdc: starting...
W20260620 19:21:35.925009 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:35 dist-test-slave-mwkb krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:36.032514 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44051
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:33911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:44051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:36.107185 22809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:36.107415 22809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:36.107455 22809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:36.108544 22809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:36.108608 22809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:36.108632 22809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:36.108659 22809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:36.108686 22809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:36.110108 22809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:33911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:44051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:44051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:36.110363 22809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:36.110565 22809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:36.115185 22817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:36.115231 22814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:36.115239 22815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:36.115510 22809 server_base.cc:1034] running on GCE node
I20260620 19:21:36.115878 22809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:36.116214 22809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:36.117347 22809 hybrid_clock.cc:648] HybridClock initialized: now 1781983296117336 us; error 30 us; skew 500 ppm
Jun 20 19:21:36 dist-test-slave-mwkb krb5kdc[22796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983296, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:36.119951 22809 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:36.120368 22809 webserver.cc:460] Webserver started at http://127.1.1.254:37989/ using document root <none> and password file <none>
I20260620 19:21:36.120543 22809 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:36.120581 22809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:36.120662 22809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:36.121716 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "86a046d04fec422484830279e3b418e1"
format_stamp: "Formatted at 2026-06-20 19:21:36 on dist-test-slave-mwkb"
server_key: "17241cfd9f94b90fcaa415b8b6ce290b"
server_key_iv: "c23394043ea9f0c82d8929cd0b8d8b4d"
server_key_version: "encryptionkey@0"
I20260620 19:21:36.121997 22809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "86a046d04fec422484830279e3b418e1"
format_stamp: "Formatted at 2026-06-20 19:21:36 on dist-test-slave-mwkb"
server_key: "17241cfd9f94b90fcaa415b8b6ce290b"
server_key_iv: "c23394043ea9f0c82d8929cd0b8d8b4d"
server_key_version: "encryptionkey@0"
I20260620 19:21:36.123178 22809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:36.123768 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:36.123914 22809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:36.123997 22809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "86a046d04fec422484830279e3b418e1"
format_stamp: "Formatted at 2026-06-20 19:21:36 on dist-test-slave-mwkb"
server_key: "17241cfd9f94b90fcaa415b8b6ce290b"
server_key_iv: "c23394043ea9f0c82d8929cd0b8d8b4d"
server_key_version: "encryptionkey@0"
I20260620 19:21:36.124068 22809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:36.127286 22809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:36.128830 22809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:36.128965 22809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:36.132560 22809 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:44051
I20260620 19:21:36.132586 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:44051 every 8 connection(s)
I20260620 19:21:36.132858 22809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:36.133700 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:36.135656 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1: Bootstrap starting.
I20260620 19:21:36.136283 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:36.136268 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22809
I20260620 19:21:36.136379 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:36.136621 1031 external_mini_cluster.cc:1428] Setting key 3d0e36d7b5be9325e08e3f929ce40321
I20260620 19:21:36.136629 22877 log.cc:824] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:36.137394 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1: No bootstrap required, opened a new log
I20260620 19:21:36.139334 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } }
I20260620 19:21:36.139474 22877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:36.139534 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a046d04fec422484830279e3b418e1, State: Initialized, Role: FOLLOWER
I20260620 19:21:36.139673 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } }
I20260620 19:21:36.139762 22877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:36.139806 22877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:36.139854 22877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:36.140554 22877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } }
I20260620 19:21:36.140666 22877 leader_election.cc:302] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a046d04fec422484830279e3b418e1; no voters:
I20260620 19:21:36.140839 22877 leader_election.cc:288] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:36.140877 22882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:36.141021 22882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [term 1 LEADER]: Becoming Leader. State: Replica: 86a046d04fec422484830279e3b418e1, State: Running, Role: LEADER
I20260620 19:21:36.141054 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:36.141124 22882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } }
I20260620 19:21:36.141435 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a046d04fec422484830279e3b418e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } } }
I20260620 19:21:36.141428 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a046d04fec422484830279e3b418e1. Latest consensus state: current_term: 1 leader_uuid: "86a046d04fec422484830279e3b418e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a046d04fec422484830279e3b418e1" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 44051 } } }
I20260620 19:21:36.141526 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:36.141534 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:36.141747 22889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:36.142140 22889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:36.142776 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:36.137028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50801 (local address 127.1.1.254:44051)
0620 19:21:36.137215 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:36.137224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:36.137256 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:21:36.137694 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:36.137700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:36.137724 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:36.137853 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:36.142320 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:36.142323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:36.142324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:36.142327 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:36.142364 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:36.142376 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:36.142433 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:36.142602 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:21:36.142636 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260620 19:21:36.143755 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:36.143780 22889 catalog_manager.cc:1347] Generated new cluster ID: 7690fdc2c8084f72a390d34b410402ba
I20260620 19:21:36.143816 22889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:36.151127 22889 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:36.151270 22889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:36.153294 22889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a046d04fec422484830279e3b418e1: Generated new TSK 0
I20260620 19:21:36.153370 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:36.145187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34861 (local address 127.1.1.254:44051)
0620 19:21:36.145217 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:36.145218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:36.150786 (+ 5568us) server_negotiation.cc:407] Connection header received
0620 19:21:36.150879 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:36.150880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:36.150885 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:36.150928 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:36.153153 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:36.153154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:36.153154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:36.153158 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:36.153172 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:36.153176 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:36.153231 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:36.153308 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:21:36.153321 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:21:36.153436 22889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:36.154745 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22809
2026-06-20T19:21:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:36 dist-test-slave-mwkb krb5kdc[22908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:36 dist-test-slave-mwkb krb5kdc[22908](info): set up 2 sockets
Jun 20 19:21:36 dist-test-slave-mwkb krb5kdc[22908](info): commencing operation
krb5kdc: starting...
W20260620 19:21:37.152654 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:37.254684 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35791
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:35791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:37.327310 22921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:37.327494 22921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:37.327538 22921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:37.328526 22921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:37.328562 22921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:37.328573 22921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:37.328584 22921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:37.328594 22921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:37.329813 22921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:35791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:35791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:37.330003 22921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:37.330129 22921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:37.334159 22926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:37.334254 22927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:37.334275 22929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:37.334193 22921 server_base.cc:1034] running on GCE node
I20260620 19:21:37.334522 22921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:37.334851 22921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:37.336006 22921 hybrid_clock.cc:648] HybridClock initialized: now 1781983297335993 us; error 38 us; skew 500 ppm
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983297, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:37.338028 22921 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:37.338392 22921 webserver.cc:460] Webserver started at http://127.1.1.254:36273/ using document root <none> and password file <none>
I20260620 19:21:37.338575 22921 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:37.338627 22921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:37.338744 22921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:37.339576 22921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "838fa248dc994eef8243b772b7504547"
format_stamp: "Formatted at 2026-06-20 19:21:37 on dist-test-slave-mwkb"
server_key: "4e48ed181e78a962897375337da15ad0"
server_key_iv: "08a69dafa904f87218a6ae129fa2d7be"
server_key_version: "encryptionkey@0"
I20260620 19:21:37.339908 22921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "838fa248dc994eef8243b772b7504547"
format_stamp: "Formatted at 2026-06-20 19:21:37 on dist-test-slave-mwkb"
server_key: "4e48ed181e78a962897375337da15ad0"
server_key_iv: "08a69dafa904f87218a6ae129fa2d7be"
server_key_version: "encryptionkey@0"
I20260620 19:21:37.341099 22921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:37.341738 22936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:37.341883 22921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:37.341956 22921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "838fa248dc994eef8243b772b7504547"
format_stamp: "Formatted at 2026-06-20 19:21:37 on dist-test-slave-mwkb"
server_key: "4e48ed181e78a962897375337da15ad0"
server_key_iv: "08a69dafa904f87218a6ae129fa2d7be"
server_key_version: "encryptionkey@0"
I20260620 19:21:37.342025 22921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:37.352643 22921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:37.354468 22921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:37.354624 22921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:37.358441 22921 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:35791
I20260620 19:21:37.358482 22988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:35791 every 8 connection(s)
I20260620 19:21:37.358803 22921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:37.359541 22989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:37.361435 22989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547: Bootstrap starting.
I20260620 19:21:37.361938 22989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:37.362205 22989 log.cc:824] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:37.362800 22989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547: No bootstrap required, opened a new log
I20260620 19:21:37.364223 22989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } }
I20260620 19:21:37.364334 22989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:37.364375 22989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838fa248dc994eef8243b772b7504547, State: Initialized, Role: FOLLOWER
I20260620 19:21:37.364517 22989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } }
I20260620 19:21:37.364575 22989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:37.364609 22989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:37.364656 22989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:37.365435 22989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } }
I20260620 19:21:37.365535 22989 leader_election.cc:302] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838fa248dc994eef8243b772b7504547; no voters:
I20260620 19:21:37.365703 22989 leader_election.cc:288] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:37.365749 22992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:37.365948 22989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:37.365952 22992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [term 1 LEADER]: Becoming Leader. State: Replica: 838fa248dc994eef8243b772b7504547, State: Running, Role: LEADER
I20260620 19:21:37.366050 22992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } }
I20260620 19:21:37.366372 22994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838fa248dc994eef8243b772b7504547. Latest consensus state: current_term: 1 leader_uuid: "838fa248dc994eef8243b772b7504547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } } }
I20260620 19:21:37.366444 22994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:37.366588 22993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838fa248dc994eef8243b772b7504547" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838fa248dc994eef8243b772b7504547" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 35791 } } }
I20260620 19:21:37.366647 22993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:37.366876 23001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:37.367249 23001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:37.368324 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 22921
I20260620 19:21:37.368415 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:37.368430 23001 catalog_manager.cc:1347] Generated new cluster ID: d69840c164dd4b17a2b16fdcbdddfd0c
I20260620 19:21:37.368472 23001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:37.368573 1031 external_mini_cluster.cc:1428] Setting key 6462c73234528348a3595f19578b70fa
I20260620 19:21:37.373323 23001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:37.374223 23001 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:37.379125 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:37.368895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41077 (local address 127.1.1.254:35791)
0620 19:21:37.369058 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:37.369082 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:37.369342 (+ 260us) server_negotiation.cc:407] Connection header received
0620 19:21:37.369454 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:37.369460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:37.369505 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:37.369615 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:37.370269 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:37.371528 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:37.372524 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:37.373210 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:37.373307 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:37.375897 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:37.375901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:37.375901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:37.375911 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:37.377678 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:37.378093 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:37.378096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:37.378096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:37.378191 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:37.378369 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:37.378370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:37.378370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:37.378587 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:37.378703 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:37.378823 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:21:37.378939 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260620 19:21:37.379850 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:37.387915 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:37.381155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45995 (local address 127.1.1.254:35791)
0620 19:21:37.381182 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:37.381183 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:37.381308 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:21:37.381354 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:37.381354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:37.381363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:37.381405 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:37.381872 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:37.382506 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:37.383332 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:37.383681 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:37.383763 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:37.384947 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:37.384949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:37.384949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:37.384955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:37.386857 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:37.387208 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:37.387209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:37.387209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:37.387278 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:37.387463 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:37.387464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:37.387464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:37.387579 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:37.387672 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:37.387764 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:21:37.387835 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:21:37.390647 23001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838fa248dc994eef8243b772b7504547: Generated new TSK 0
I20260620 19:21:37.390807 23001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:37.395401 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 22921
2026-06-20T19:21:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1240 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[23020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[23020](info): set up 2 sockets
Jun 20 19:21:37 dist-test-slave-mwkb krb5kdc[23020](info): commencing operation
krb5kdc: starting...
W20260620 19:21:38.352823 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:38 dist-test-slave-mwkb krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:38.459297 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46439
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:38.535333 23033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:38.535513 23033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:38.535538 23033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:38.536578 23033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:38.536615 23033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:38.536628 23033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:38.536666 23033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:38.536684 23033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:38.537967 23033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:38.538225 23033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:38.538412 23033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:38.542544 23039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:38.542608 23038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:38.542655 23041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:38.542707 23033 server_base.cc:1034] running on GCE node
I20260620 19:21:38.542920 23033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:38.543174 23033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:38.544327 23033 hybrid_clock.cc:648] HybridClock initialized: now 1781983298544315 us; error 44 us; skew 500 ppm
Jun 20 19:21:38 dist-test-slave-mwkb krb5kdc[23020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983298, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:38.546655 23033 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:38.547071 23033 webserver.cc:460] Webserver started at http://127.1.1.254:33557/ using document root <none> and password file <none>
I20260620 19:21:38.547269 23033 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:38.547329 23033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:38.547468 23033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:38.548388 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "2ea62caec08a4e3986d2b46d45083337"
format_stamp: "Formatted at 2026-06-20 19:21:38 on dist-test-slave-mwkb"
server_key: "89e579227e9468e9b2a70f1f3291d2fd"
server_key_iv: "ef03737f8ee60cefb7afe3bb28356a19"
server_key_version: "encryptionkey@0"
I20260620 19:21:38.548699 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "2ea62caec08a4e3986d2b46d45083337"
format_stamp: "Formatted at 2026-06-20 19:21:38 on dist-test-slave-mwkb"
server_key: "89e579227e9468e9b2a70f1f3291d2fd"
server_key_iv: "ef03737f8ee60cefb7afe3bb28356a19"
server_key_version: "encryptionkey@0"
I20260620 19:21:38.549912 23033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:21:38.550503 23048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:38.550645 23033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:38.550730 23033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "2ea62caec08a4e3986d2b46d45083337"
format_stamp: "Formatted at 2026-06-20 19:21:38 on dist-test-slave-mwkb"
server_key: "89e579227e9468e9b2a70f1f3291d2fd"
server_key_iv: "ef03737f8ee60cefb7afe3bb28356a19"
server_key_version: "encryptionkey@0"
I20260620 19:21:38.550797 23033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:38.566102 23033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:38.567687 23033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:38.567847 23033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:38.571321 23033 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46439
I20260620 19:21:38.571348 23100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46439 every 8 connection(s)
I20260620 19:21:38.571640 23033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:38.572324 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:38.572965 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23033
I20260620 19:21:38.573081 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:38.573259 1031 external_mini_cluster.cc:1428] Setting key a3cf530854be42c3988d253518bbf8d7
I20260620 19:21:38.575132 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337: Bootstrap starting.
I20260620 19:21:38.575757 23101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:38.576033 23101 log.cc:824] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:38.576591 23101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337: No bootstrap required, opened a new log
I20260620 19:21:38.578415 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:38.573660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56751 (local address 127.1.1.254:46439)
0620 19:21:38.573804 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:38.573813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:38.573838 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:21:38.574328 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:38.574336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:38.574401 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:38.574525 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:38.574672 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:38.576011 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:38.576739 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:38.577163 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:38.577265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:38.577890 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:38.577894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:38.577894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:38.578029 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:38.578053 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:38.578063 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:38.578131 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:38.578212 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:21:38.578282 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260620 19:21:38.578367 23101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } }
I20260620 19:21:38.578478 23101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:38.578522 23101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ea62caec08a4e3986d2b46d45083337, State: Initialized, Role: FOLLOWER
I20260620 19:21:38.578606 23101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } }
I20260620 19:21:38.578678 23101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:38.578708 23101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:38.578760 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:38.579363 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:38.579371 23101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } }
I20260620 19:21:38.579468 23101 leader_election.cc:302] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ea62caec08a4e3986d2b46d45083337; no voters:
I20260620 19:21:38.579646 23101 leader_election.cc:288] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:38.579674 23106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:38.579787 23106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [term 1 LEADER]: Becoming Leader. State: Replica: 2ea62caec08a4e3986d2b46d45083337, State: Running, Role: LEADER
I20260620 19:21:38.579881 23101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:38.579870 23106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } }
I20260620 19:21:38.580199 23107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ea62caec08a4e3986d2b46d45083337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } } }
I20260620 19:21:38.580287 23107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:38.580510 23108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ea62caec08a4e3986d2b46d45083337. Latest consensus state: current_term: 1 leader_uuid: "2ea62caec08a4e3986d2b46d45083337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ea62caec08a4e3986d2b46d45083337" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46439 } } }
I20260620 19:21:38.580622 23108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:38.580790 23113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:38.581284 23113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:38.582588 23113 catalog_manager.cc:1347] Generated new cluster ID: 2f04a806a29148cfbabb6355d46c20c2
I20260620 19:21:38.582643 23113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:38.584265 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:38.581381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50083 (local address 127.1.1.254:46439)
0620 19:21:38.581410 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:38.581412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:38.581536 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:21:38.581620 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:38.581621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:38.581631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:38.581657 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:38.581815 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:38.582296 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:38.583066 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:38.583326 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:38.583390 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:38.584028 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:38.584029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:38.584030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:38.584035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:38.584048 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:38.584052 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:38.584111 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:38.584167 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:21:38.584187 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:21:38.594386 23113 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:38.594956 23113 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:38.606472 23113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ea62caec08a4e3986d2b46d45083337: Generated new TSK 0
I20260620 19:21:38.606654 23113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:38.610685 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23033
2026-06-20T19:21:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:38 dist-test-slave-mwkb krb5kdc[23132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:38 dist-test-slave-mwkb krb5kdc[23132](info): set up 2 sockets
Jun 20 19:21:38 dist-test-slave-mwkb krb5kdc[23132](info): commencing operation
krb5kdc: starting...
W20260620 19:21:39.616833 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:39 dist-test-slave-mwkb krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:39.719595 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46825
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:46825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:39.794703 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:39.794974 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:39.795013 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:39.796268 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:39.796332 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:39.796358 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:39.796396 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:39.796419 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:39.797963 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:46825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:39.798255 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:39.798674 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:39.803591 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:39.803599 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:39.803658 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:39.803817 23145 server_base.cc:1034] running on GCE node
I20260620 19:21:39.804035 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:39.804319 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:39.805452 23145 hybrid_clock.cc:648] HybridClock initialized: now 1781983299805442 us; error 26 us; skew 500 ppm
Jun 20 19:21:39 dist-test-slave-mwkb krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983299, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:39.807721 23145 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:39.808110 23145 webserver.cc:460] Webserver started at http://127.1.1.254:41175/ using document root <none> and password file <none>
I20260620 19:21:39.808332 23145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:39.808394 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:39.808545 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:39.809661 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "62acf35586994a419325f1711b1e07ea"
format_stamp: "Formatted at 2026-06-20 19:21:39 on dist-test-slave-mwkb"
server_key: "15d7c2b71875aefcfc06239e56ee96e0"
server_key_iv: "0ffe27987672b59a1b5a4e7631c75b0a"
server_key_version: "encryptionkey@0"
I20260620 19:21:39.809988 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "62acf35586994a419325f1711b1e07ea"
format_stamp: "Formatted at 2026-06-20 19:21:39 on dist-test-slave-mwkb"
server_key: "15d7c2b71875aefcfc06239e56ee96e0"
server_key_iv: "0ffe27987672b59a1b5a4e7631c75b0a"
server_key_version: "encryptionkey@0"
I20260620 19:21:39.811177 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:39.811932 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:39.812084 23145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:39.812172 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "62acf35586994a419325f1711b1e07ea"
format_stamp: "Formatted at 2026-06-20 19:21:39 on dist-test-slave-mwkb"
server_key: "15d7c2b71875aefcfc06239e56ee96e0"
server_key_iv: "0ffe27987672b59a1b5a4e7631c75b0a"
server_key_version: "encryptionkey@0"
I20260620 19:21:39.812245 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:39.815346 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:39.816958 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:39.817138 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:39.820773 23145 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:46825
I20260620 19:21:39.820806 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:46825 every 8 connection(s)
I20260620 19:21:39.821166 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:39.822067 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:39.823315 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23145
I20260620 19:21:39.823418 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:39.823594 1031 external_mini_cluster.cc:1428] Setting key 3ffde89d325f84d6d62c09b47cc4bcca
I20260620 19:21:39.824565 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea: Bootstrap starting.
I20260620 19:21:39.825384 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:39.825730 23213 log.cc:824] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:39.825922 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:39.824031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33069 (local address 127.1.1.254:46825)
0620 19:21:39.824209 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:39.824217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:39.824244 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:21:39.824684 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:39.824693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:39.824724 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:39.824838 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:39.825496 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:39.825499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:39.825499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:39.825504 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:39.825546 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:39.825561 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:39.825624 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:39.825702 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:21:39.825730 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260620 19:21:39.826393 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea: No bootstrap required, opened a new log
I20260620 19:21:39.826790 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:39.828455 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } }
I20260620 19:21:39.828621 23213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:21:39.828677 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62acf35586994a419325f1711b1e07ea, State: Initialized, Role: FOLLOWER
I20260620 19:21:39.828794 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } }
I20260620 19:21:39.828903 23213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:21:39.828981 23213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:21:39.829046 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:39.829047 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:39.828135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39891 (local address 127.1.1.254:46825)
0620 19:21:39.828175 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:39.828177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:39.828234 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:21:39.828298 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:39.828299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:39.828303 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:39.828358 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:39.828834 (+ 476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:39.828835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:39.828835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:21:39.828851 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:39.828862 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:21:39.828866 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:39.828921 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:39.828967 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:21:39.828978 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:21:39.829666 23213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } }
I20260620 19:21:39.829744 23213 leader_election.cc:302] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62acf35586994a419325f1711b1e07ea; no voters:
I20260620 19:21:39.829845 23213 leader_election.cc:288] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:21:39.829900 23223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:39.830040 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:39.830044 23223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [term 1 LEADER]: Becoming Leader. State: Replica: 62acf35586994a419325f1711b1e07ea, State: Running, Role: LEADER
I20260620 19:21:39.830157 23223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } }
I20260620 19:21:39.830473 23224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62acf35586994a419325f1711b1e07ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } } }
I20260620 19:21:39.830574 23224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:39.830736 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62acf35586994a419325f1711b1e07ea. Latest consensus state: current_term: 1 leader_uuid: "62acf35586994a419325f1711b1e07ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62acf35586994a419325f1711b1e07ea" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 46825 } } }
I20260620 19:21:39.830791 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:39.831100 23232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:39.831488 23232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:39.832823 23232 catalog_manager.cc:1347] Generated new cluster ID: 05832682c12844bab17b72849603b411
I20260620 19:21:39.832859 23232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:39.863585 23232 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:39.863735 23232 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:39.867584 23232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62acf35586994a419325f1711b1e07ea: Generated new TSK 0
I20260620 19:21:39.867741 23232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:39.874800 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23145
2026-06-20T19:21:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1264 ms)
[----------] 18 tests from AuthTokenIssuingTest (22078 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:39 dist-test-slave-mwkb krb5kdc[23244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:39 dist-test-slave-mwkb krb5kdc[23244](info): set up 2 sockets
Jun 20 19:21:39 dist-test-slave-mwkb krb5kdc[23244](info): commencing operation
krb5kdc: starting...
W20260620 19:21:40.880903 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:40 dist-test-slave-mwkb krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:40.986002 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41719
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:41.061844 23257 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:41.062031 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:41.062047 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:41.062067 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:41.063092 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:41.063126 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:41.063140 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:41.063153 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:41.063165 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:41.064529 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:41.064699 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:41.064850 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:41.069067 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:41.069118 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:41.069073 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:41.069183 23257 server_base.cc:1034] running on GCE node
I20260620 19:21:41.069475 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:41.069700 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:41.070842 23257 hybrid_clock.cc:648] HybridClock initialized: now 1781983301070833 us; error 25 us; skew 500 ppm
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:41.072981 23257 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:41.073365 23257 webserver.cc:460] Webserver started at http://127.1.1.254:46743/ using document root <none> and password file <none>
I20260620 19:21:41.073549 23257 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:41.073582 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:41.073664 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:41.074467 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "9ddfcc3ca60e40678b523949c7ed0f36"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "ac027d9783517ba81219e5dbd8b0f1ee"
server_key_iv: "09cd2c7a769f0cc5a4cdc57190569076"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.074733 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "9ddfcc3ca60e40678b523949c7ed0f36"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "ac027d9783517ba81219e5dbd8b0f1ee"
server_key_iv: "09cd2c7a769f0cc5a4cdc57190569076"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.075848 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:21:41.076452 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:41.076572 23257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:41.076624 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "9ddfcc3ca60e40678b523949c7ed0f36"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "ac027d9783517ba81219e5dbd8b0f1ee"
server_key_iv: "09cd2c7a769f0cc5a4cdc57190569076"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.076709 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:41.086344 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:41.088174 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:41.088325 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:41.092839 23257 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:41719
I20260620 19:21:41.093142 23324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:41719 every 8 connection(s)
I20260620 19:21:41.093321 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:41.094281 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:41.096416 23325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.100222 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23257
I20260620 19:21:41.100344 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:41.100514 1031 external_mini_cluster.cc:1428] Setting key 862857bda97b51823833cff1f29adbc4
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:41.106554 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.097419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41719 (local address 127.0.0.1:59112)
0620 19:21:41.097727 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.097785 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.098513 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.098823 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.098825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.099108 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.099251 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.099255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.100288 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.100289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.101048 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.101050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.101573 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.101575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.101637 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.102144 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.102151 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.103409 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.105158 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.105164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.105166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.105350 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.105599 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.105600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.105601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.105684 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.106000 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.106004 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.106060 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.106169 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.106384 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":85,"thread_start_us":110,"threads_started":1}
I20260620 19:21:41.106559 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.097511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.1.1.254:41719)
0620 19:21:41.097746 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.097760 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.098001 (+ 241us) server_negotiation.cc:407] Connection header received
0620 19:21:41.098693 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.098695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.098721 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.098817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.099356 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.100174 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.101163 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.101464 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.101593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.103499 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.103505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.103506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.103512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.105023 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.105441 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.105443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.105444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.105520 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.105745 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.105746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.105747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.105934 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.106021 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.106192 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.106324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.1.1.253@KRBTEST.COM; defaulting to no policy
I20260620 19:21:41.107188 23325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.107525 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.107431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33114)
0620 19:21:41.107469 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.107509 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:21:41.107657 23276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
I20260620 19:21:41.108100 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.108004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33116)
0620 19:21:41.108046 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.108078 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:21:41.108203 23325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:36509: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
I20260620 19:21:41.121888 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:36509
--webserver_interface=127.1.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:21:41.157649 23325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } attempt: 1
I20260620 19:21:41.158082 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.157956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33118)
0620 19:21:41.158014 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.158050 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:21:41.158465 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.158391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33134)
0620 19:21:41.158428 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.158450 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 19:21:41.158581 23325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:36509: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
W20260620 19:21:41.199833 23332 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:41.200021 23332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:41.200037 23332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:41.200057 23332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:41.201224 23332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:41.201263 23332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:41.201277 23332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:41.201306 23332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:41.201332 23332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:41.202756 23332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:36509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:41.202984 23332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:41.203194 23332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:41.207855 23338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:41.207927 23340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:41.207885 23337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:41.207904 23332 server_base.cc:1034] running on GCE node
I20260620 19:21:41.208392 23332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:41.208658 23332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:41.209810 23332 hybrid_clock.cc:648] HybridClock initialized: now 1781983301209802 us; error 33 us; skew 500 ppm
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:41.215987 23332 init.cc:377] Logged in from keytab as kudu/127.1.1.253@KRBTEST.COM (short username kudu)
I20260620 19:21:41.216388 23332 webserver.cc:460] Webserver started at http://127.1.1.253:35135/ using document root <none> and password file <none>
I20260620 19:21:41.216591 23332 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:41.216665 23332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:41.216796 23332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:41.217787 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data/instance:
uuid: "696698c6be2a4f68a0ebf85ca2eab25a"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "489de0e6f855cc2df3d2b1150376f9d4"
server_key_iv: "34a7bee5a4240b3567a3cf787fa61ef1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.218111 23332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal/instance:
uuid: "696698c6be2a4f68a0ebf85ca2eab25a"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "489de0e6f855cc2df3d2b1150376f9d4"
server_key_iv: "34a7bee5a4240b3567a3cf787fa61ef1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.219317 23332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:41.220009 23347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:41.220198 23332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:41.220293 23332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "696698c6be2a4f68a0ebf85ca2eab25a"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "489de0e6f855cc2df3d2b1150376f9d4"
server_key_iv: "34a7bee5a4240b3567a3cf787fa61ef1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.220361 23332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:41.226715 23325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } attempt: 2
I20260620 19:21:41.227064 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.226958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33144)
0620 19:21:41.227007 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.227039 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:21:41.227362 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.227295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33150)
0620 19:21:41.227329 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.227348 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:21:41.227458 23325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:36509: Network error: Client connection negotiation failed: client connection to 127.1.1.253:36509: connect: Connection refused (error 111)
I20260620 19:21:41.238308 23332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:41.240201 23332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:41.240340 23332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:41.243769 23332 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.253:36509
I20260620 19:21:41.243809 23399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.253:36509 every 8 connection(s)
I20260620 19:21:41.244060 23332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
I20260620 19:21:41.244827 23400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:41.245728 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23332
I20260620 19:21:41.245819 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-1/wal/instance
I20260620 19:21:41.245981 1031 external_mini_cluster.cc:1428] Setting key 62b7caccd27fe607d9f89b3f295cd3fe
I20260620 19:21:41.246371 23400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.1.252@KRBTEST.COM; defaulting to no policy
I20260620 19:21:41.254905 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.246984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.1.1.254:41719)
0620 19:21:41.247031 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.247033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.247278 (+ 245us) server_negotiation.cc:407] Connection header received
0620 19:21:41.247855 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.247856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.247865 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.247902 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.248528 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.249358 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.250298 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.250633 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.250707 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.252496 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.252497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.252498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.252505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.253803 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.254208 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.254209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.254210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.254267 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.254459 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.254459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.254459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.254550 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.254596 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.254800 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.254855 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:21:41.255125 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.246950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41719 (local address 127.0.0.1:59124)
0620 19:21:41.247142 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.247184 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.247767 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.247963 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.247967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.248280 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.248449 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.248453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.249444 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.249445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.250216 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.250218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.250722 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.250723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.250850 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.251298 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.251305 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.252416 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.253867 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.253885 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.253886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.254146 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.254325 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.254325 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.254325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.254411 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.254601 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.254605 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.254709 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.254778 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.254870 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.252@KRBTEST.COM" created.
I20260620 19:21:41.255654 23400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:41.263257 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.255870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33154)
0620 19:21:41.256050 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.256060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.256111 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.256290 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.256291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.256403 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.256435 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.256436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.257722 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.257723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.258198 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.258199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.258667 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.258668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.258729 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.259106 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.259111 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.260499 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.262179 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.262181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.262181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.262390 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.262586 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.262587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.262587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.262654 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.262898 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.262899 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.262950 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.263034 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.263142 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:21:41.263280 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.255931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.1.1.253:36509)
0620 19:21:41.256062 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.256066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.256110 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:21:41.256167 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.256169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.256181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.256277 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.256503 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.257657 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.258287 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.258586 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.258667 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.260587 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.260593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.260593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.260599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.262095 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.262471 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.262473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.262473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.262521 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.262716 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.262717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.262717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.262832 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.262890 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.263054 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.263185 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260620 19:21:41.263933 23400 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.264247 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.264128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39206)
0620 19:21:41.264179 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.264231 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 19:21:41.264369 23349 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
I20260620 19:21:41.264715 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.264639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39220)
0620 19:21:41.264678 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.264700 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:21:41.264809 23400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:44533: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
I20260620 19:21:41.269294 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:44533
--webserver_interface=127.1.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:42121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:21:41.314013 23400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } attempt: 1
I20260620 19:21:41.314468 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.314354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39228)
0620 19:21:41.314407 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.314441 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:21:41.314795 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.314706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39244)
0620 19:21:41.314750 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.314773 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:21:41.314879 23400 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:44533: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
I20260620 19:21:41.318629 23325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } attempt: 3
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:41.324945 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.318959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.1.1.253:36509)
0620 19:21:41.319019 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.319021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.319031 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:21:41.319077 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.319078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.319089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.319154 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.319451 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.319961 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.320635 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.320868 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.320934 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.322544 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.322546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.322546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.322553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.323896 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.324221 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.324222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.324223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.324274 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.324447 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.324447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.324448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.324581 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.324662 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.324799 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.324864 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:21:41.324950 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.318889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33156)
0620 19:21:41.318931 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.318958 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.319019 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.319171 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.319172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.319317 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.319373 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.319375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.320041 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.320042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.320547 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.320549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.320933 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.320933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.321002 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.321429 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.321438 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.322470 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.323976 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.323977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.323977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.324148 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.324338 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.324339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.324339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.324389 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.324656 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.324657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.324699 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.324776 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.324863 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:21:41.325412 23325 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.325709 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.325609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39252)
0620 19:21:41.325651 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.325683 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:21:41.326050 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.325969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39268)
0620 19:21:41.326011 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.326033 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:21:41.326146 23325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:44533: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
W20260620 19:21:41.346477 23407 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:41.346652 23407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:41.346688 23407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:41.346727 23407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:41.347724 23407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:41.347759 23407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:41.347772 23407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:41.347817 23407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:41.347844 23407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:41.349109 23407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:42121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41719,127.1.1.253:36509,127.1.1.252:44533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:44533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:41.349304 23407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:41.349486 23407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:41.353279 23413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:41.353367 23407 server_base.cc:1034] running on GCE node
W20260620 19:21:41.353482 23415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:41.353384 23412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:41.353739 23407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:41.354012 23407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:41.355176 23407 hybrid_clock.cc:648] HybridClock initialized: now 1781983301355167 us; error 28 us; skew 500 ppm
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:41.357276 23325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } attempt: 1
I20260620 19:21:41.357438 23407 init.cc:377] Logged in from keytab as kudu/127.1.1.252@KRBTEST.COM (short username kudu)
I20260620 19:21:41.357583 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.357463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39272)
0620 19:21:41.357510 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.357553 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:21:41.357903 23407 webserver.cc:460] Webserver started at http://127.1.1.252:39847/ using document root <none> and password file <none>
I20260620 19:21:41.357908 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.357800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39274)
0620 19:21:41.357845 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.357869 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:21:41.358027 23325 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:44533: Network error: Client connection negotiation failed: client connection to 127.1.1.252:44533: connect: Connection refused (error 111)
I20260620 19:21:41.358158 23407 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:41.358227 23407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:41.358355 23407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:41.359318 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data/instance:
uuid: "0472e8f41e2046f7ae22b7596a632b3b"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "2bbf695817e674b48079531da06df50d"
server_key_iv: "9f97cc57762d6951b034163816e9bed1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.359671 23407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal/instance:
uuid: "0472e8f41e2046f7ae22b7596a632b3b"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "2bbf695817e674b48079531da06df50d"
server_key_iv: "9f97cc57762d6951b034163816e9bed1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.360905 23407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:41.361672 23422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:41.361841 23407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:41.361931 23407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "0472e8f41e2046f7ae22b7596a632b3b"
format_stamp: "Formatted at 2026-06-20 19:21:41 on dist-test-slave-mwkb"
server_key: "2bbf695817e674b48079531da06df50d"
server_key_iv: "9f97cc57762d6951b034163816e9bed1"
server_key_version: "encryptionkey@0"
I20260620 19:21:41.362010 23407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:41.373823 23407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:41.375226 23407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:41.375371 23407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:41.378993 23407 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.252:44533
I20260620 19:21:41.379022 23474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.252:44533 every 8 connection(s)
I20260620 19:21:41.379369 23407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
I20260620 19:21:41.380301 23475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:41.381546 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.383011 23400 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } attempt: 2
I20260620 19:21:41.383781 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23407
I20260620 19:21:41.383862 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781983021660275-1031-0/minicluster-data/master-2/wal/instance
I20260620 19:21:41.384028 1031 external_mini_cluster.cc:1428] Setting key 019543723dcc5e9eaa5379378a47df27
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:21:41.391428 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.382026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.1.1.254:41719)
0620 19:21:41.382076 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.382078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.382242 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:21:41.382668 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.382669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.382679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.382722 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.383289 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.384010 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.385276 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.385535 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.385603 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.387916 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.387918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.387918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.387925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.390017 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390491 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.390492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.390493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.390554 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390844 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.390845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.390846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.390981 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.391051 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.391263 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.391340 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:21:41.391644 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.382018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41719 (local address 127.0.0.1:59130)
0620 19:21:41.382154 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.382180 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.382558 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.382788 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.382791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.383002 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.383206 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.383210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.384084 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.384084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.385162 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.385164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.385642 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.385643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.385774 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.386263 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.386270 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.387833 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.390117 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390125 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.390127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.390385 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.390638 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.390639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.390758 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.391052 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.391057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.391113 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.391234 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.391375 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260620 19:21:41.391742 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.383243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39286)
0620 19:21:41.383292 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.383301 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.383337 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.383629 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.383630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.383772 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.383808 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.383809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.385221 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.385222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.385978 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.385981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.386589 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.386590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.386655 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.387055 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.387060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.388377 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.390223 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.390225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.390392 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.390680 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.390681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.390740 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.391173 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.391174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.391222 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.391295 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.391394 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:21:41.392166 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.392329 23478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.383321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.1.1.252:44533)
0620 19:21:41.383471 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.383474 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.383488 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:21:41.383530 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.383531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.383545 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.383622 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.383913 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.385137 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.386067 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.386501 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.386571 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.388458 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.388465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.388466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.388473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.390105 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390488 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.390496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.390496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.390608 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.390812 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.390813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.390813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.391101 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.391192 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.392060 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.392147 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260620 19:21:41.393811 23400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: Bootstrap starting.
I20260620 19:21:41.394068 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.384368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.1.1.252:44533)
0620 19:21:41.384645 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.384647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.384652 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:21:41.384666 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.384667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.384674 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.384737 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.385439 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.385971 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.386954 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.387280 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.387351 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.390569 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.390570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.390570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.390576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.392954 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.393305 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.393306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.393306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.393359 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.393555 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.393556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.393556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.393657 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.393785 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.393873 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.393944 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":76,"threads_started":1}
I20260620 19:21:41.394730 23400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:41.395007 23400 log.cc:824] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:41.395828 23400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: No bootstrap required, opened a new log
I20260620 19:21:41.397629 23400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.397810 23400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696698c6be2a4f68a0ebf85ca2eab25a, State: Initialized, Role: FOLLOWER
I20260620 19:21:41.397910 23400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.398175 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.398248 23400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:41.398351 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
W20260620 19:21:41.400516 23493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:41.400609 23493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:41.402159 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.392411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:36509 (local address 127.0.0.1:33170)
0620 19:21:41.392574 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.392583 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.392682 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.392819 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.392820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.395374 (+ 2554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.395428 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.395431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.396368 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.396369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.396853 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.396854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.397339 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.397340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.397410 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.397794 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.397799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.399200 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.401041 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.401042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.401042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.401281 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.401491 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.401492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.401493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.401550 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.401863 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.401864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.401905 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.401987 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.402072 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260620 19:21:41.402161 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.392530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.1.1.253:36509)
0620 19:21:41.392577 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.392586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.392660 (+ 74us) server_negotiation.cc:407] Connection header received
0620 19:21:41.392763 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.392764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.392773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.392818 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.395551 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.396300 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.396930 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.397228 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.397288 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.399307 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.399308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.399308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.399316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.400961 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.401364 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.401365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.401365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.401418 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.401644 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.401645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.401645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.401802 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.401859 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.402011 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.402076 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:21:41.400384 23332 master_runner.cc:186] Error getting master registration for 127.1.1.254:41719: OK, instance_id { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" instance_seqno: 1781983301091408 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:21:41.403192 23332 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:21:41.403548 23475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:41.404057 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.395631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.1.1.254:41719)
0620 19:21:41.395671 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.395673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.395680 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:41.395721 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.395722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.395731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.395786 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.396311 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.396986 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.397898 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.398204 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.398248 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.400879 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.400881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.400881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.400888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.402829 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.403186 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.403187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.403188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.403237 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.403425 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.403426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.403427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.403545 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.403679 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.403881 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.403957 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:41.411057 23477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.403765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39298)
0620 19:21:41.403805 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.403814 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.403846 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.403966 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.403967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.404104 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.404138 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.404139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.404813 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.404814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.405469 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.405471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.406439 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.406440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.406492 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.406862 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.406867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.408093 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.409954 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.409955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.409956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.410177 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.410376 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.410377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.410378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.410440 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.410729 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.410730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.410777 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.410846 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.410942 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:21:41.411139 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.403815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.1.1.252:44533)
0620 19:21:41.403846 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.403848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.403856 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:41.403897 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.403898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.403907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.403965 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.404216 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.404688 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.405529 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.405828 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.405895 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.408189 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.408191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.408191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.408198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.409862 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.410246 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.410247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.410248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.410302 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.410510 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.410510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.410511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.410663 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.410736 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.410897 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.411002 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":17}
I20260620 19:21:41.412357 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: Bootstrap starting.
I20260620 19:21:41.412734 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.405335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33172 (local address 127.1.1.253:36509)
0620 19:21:41.405367 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.405368 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.405375 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:41.405393 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.405394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.405401 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.405456 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.405920 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.406341 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.407223 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.407515 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.407588 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.409796 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.409798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.409798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.409805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.411457 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.411828 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.411829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.411830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.411880 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.412108 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.412109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.412109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.412265 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.412375 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.412578 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.412651 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:21:41.412966 23475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:41.413345 23475 log.cc:824] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:41.414004 23475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: No bootstrap required, opened a new log
I20260620 19:21:41.415350 23475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.415459 23475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0472e8f41e2046f7ae22b7596a632b3b, State: Initialized, Role: FOLLOWER
I20260620 19:21:41.415535 23475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.415820 23475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:41.415784 23496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.415901 23496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.417687 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 19:21:41.417721 23507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:41.417846 23507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:41.417687 23407 master_runner.cc:186] Error getting master registration for 127.1.1.254:41719: OK, instance_id { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" instance_seqno: 1781983301091408 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:21:41.418877 23299 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9ddfcc3ca60e40678b523949c7ed0f36"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59144
I20260620 19:21:41.419806 23407 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:21:41.433122 23325 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } attempt: 2
Jun 20 19:21:41 dist-test-slave-mwkb krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983301, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:21:41.440441 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.433325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:44533 (local address 127.0.0.1:39306)
0620 19:21:41.433363 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:41.433372 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:41.433402 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:41.433561 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:41.433562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:41.433734 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:41.433771 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.433772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.434586 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.434587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.435432 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:41.435435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.435909 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.435911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:41.435969 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.436400 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:41.436404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:41.437793 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:41.439441 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.439442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.439443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.439681 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.439878 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:41.439879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:41.439879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:41.439948 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:41.440173 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:41.440174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:41.440211 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:21:41.440275 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:21:41.440354 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:21:41.440472 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.433380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39306 (local address 127.1.1.252:44533)
0620 19:21:41.433420 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.433421 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.433445 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:21:41.433478 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.433478 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.433486 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.433528 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.433843 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.434508 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.435524 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.435818 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.435893 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.437881 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.437882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.437882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.437888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.439335 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.439767 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.439768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.439768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.439821 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.439993 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.439994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.439994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.440116 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.440173 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.440306 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.440375 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:21:41.442116 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: Bootstrap starting.
I20260620 19:21:41.442721 23325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:41.442976 23325 log.cc:824] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:41.443673 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: No bootstrap required, opened a new log
I20260620 19:21:41.444763 23325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.444860 23325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ddfcc3ca60e40678b523949c7ed0f36, State: Initialized, Role: FOLLOWER
I20260620 19:21:41.444916 23325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.445199 23510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.445329 23510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.445369 23299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:21:41.445458 23299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:41.446125 23299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.446347 23299 leader_election.cc:288] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [CANDIDATE]: Term 1 election: Requested vote from peers 696698c6be2a4f68a0ebf85ca2eab25a (127.1.1.253:36509), 0472e8f41e2046f7ae22b7596a632b3b (127.1.1.252:44533)
I20260620 19:21:41.446610 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "696698c6be2a4f68a0ebf85ca2eab25a"
I20260620 19:21:41.446710 23374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:41.447413 23374 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddfcc3ca60e40678b523949c7ed0f36 in term 1.
I20260620 19:21:41.447582 23276 leader_election.cc:302] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 696698c6be2a4f68a0ebf85ca2eab25a, 9ddfcc3ca60e40678b523949c7ed0f36; no voters:
I20260620 19:21:41.447868 23449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0472e8f41e2046f7ae22b7596a632b3b"
I20260620 19:21:41.447973 23449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:41.448648 23449 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ddfcc3ca60e40678b523949c7ed0f36 in term 1.
I20260620 19:21:41.445366 23325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:41.447695 23510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:41.450090 23510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [term 1 LEADER]: Becoming Leader. State: Replica: 9ddfcc3ca60e40678b523949c7ed0f36, State: Running, Role: LEADER
I20260620 19:21:41.450172 23510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } }
I20260620 19:21:41.450941 23510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ddfcc3ca60e40678b523949c7ed0f36. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.451005 23510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: This master's current role is: LEADER
W20260620 19:21:41.451418 23530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:41.451458 23530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:41.451668 23531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:41.451795 23531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:41.464623 23449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [term 1 FOLLOWER]: Refusing update from remote peer 9ddfcc3ca60e40678b523949c7ed0f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:41.464938 23510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:41.466173 23496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ddfcc3ca60e40678b523949c7ed0f36. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.466253 23496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.466591 23374 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [term 1 FOLLOWER]: Refusing update from remote peer 9ddfcc3ca60e40678b523949c7ed0f36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:41.466814 23510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:41.467834 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ddfcc3ca60e40678b523949c7ed0f36. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.467921 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.468816 23533 mvcc.cc:205] Tried to move back new op lower bound from 7299003602796613632 to 7299003602740502528. Current Snapshot: MvccSnapshot[applied={T|T < 7299003602796613632}]
I20260620 19:21:41.469089 23510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.469161 23510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:41.469444 23496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.469509 23496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.470044 23534 mvcc.cc:205] Tried to move back new op lower bound from 7299003602796613632 to 7299003602740502528. Current Snapshot: MvccSnapshot[applied={T|T < 7299003602796613632}]
I20260620 19:21:41.470466 23531 catalog_manager.cc:1347] Generated new cluster ID: a62d669f8d444e42a65c8e0fa4621716
I20260620 19:21:41.470518 23531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:41.476557 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.449567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.1.1.252:44533)
0620 19:21:41.449592 (+ 25us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.449593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.457294 (+ 7701us) server_negotiation.cc:407] Connection header received
0620 19:21:41.457389 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.457390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.457399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.457428 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.458159 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.458872 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.459930 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.460230 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.460269 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.471251 (+ 10982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.471253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.471253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.471260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.472788 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.475161 (+ 2373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.475163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.475163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.475215 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.475637 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.475638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.475639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.475750 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.475828 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.476416 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.476486 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:21:41.476639 23531 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:41.477278 23531 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:41.477957 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.449422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.1.1.254:41719)
0620 19:21:41.449451 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.449453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.460889 (+ 11436us) server_negotiation.cc:407] Connection header received
0620 19:21:41.460981 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.460982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.460991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.461020 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.461770 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.462495 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.463497 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.463801 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.463839 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.473188 (+ 9349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.473189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.473189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.473196 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.474726 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.475398 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.475399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.475399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.475443 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.475999 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.476000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.476000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.476130 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.476224 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.477786 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.477874 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:21:41.482401 23531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36: Generated new TSK 0
I20260620 19:21:41.482542 23531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:41.483844 23528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.483927 23528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ddfcc3ca60e40678b523949c7ed0f36 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:41.485476 23482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ddfcc3ca60e40678b523949c7ed0f36" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41719 } } peers { permanent_uuid: "696698c6be2a4f68a0ebf85ca2eab25a" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 36509 } } peers { permanent_uuid: "0472e8f41e2046f7ae22b7596a632b3b" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 44533 } } }
I20260620 19:21:41.485589 23482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:41.485980 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:41.449275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.1.1.253:36509)
0620 19:21:41.449304 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.449306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.464574 (+ 15268us) server_negotiation.cc:407] Connection header received
0620 19:21:41.464819 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.464821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.464830 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.464859 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.468954 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.469694 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.476880 (+ 7186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:41.477190 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:41.477231 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:41.480426 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:41.480428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:41.480428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:41.480434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:41.482043 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.482907 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.482908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.482908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.482960 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:41.483165 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:41.483166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:41.483166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:41.483271 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:41.483329 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:41.485814 (+ 2485us) server_negotiation.cc:299] Negotiation successful
0620 19:21:41.485892 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 19:21:41.510923 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.509540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.1.1.253:36509)
0620 19:21:41.509588 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.509589 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.510361 (+ 772us) server_negotiation.cc:407] Connection header received
0620 19:21:41.510450 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.510451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.510460 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.510486 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.510872 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33180: BlockingRecv error: recv got EOF from 127.0.0.1:33180 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 19:21:41.511571 23328 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.509859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.1.1.254:41719)
0620 19:21:41.509895 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.509896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.511128 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 19:21:41.511209 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.511210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.511219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.511244 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.511508 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59172: BlockingRecv error: recv got EOF from 127.0.0.1:59172 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:41.512187 23480 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.510122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39334 (local address 127.1.1.252:44533)
0620 19:21:41.510154 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.510156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.511758 (+ 1602us) server_negotiation.cc:407] Connection header received
0620 19:21:41.511838 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.511839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.511847 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.511871 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.512123 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39334: BlockingRecv error: recv got EOF from 127.0.0.1:39334 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:21:41.514313 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.513549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.1.1.253:36509)
0620 19:21:41.513608 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.513610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.513844 (+ 234us) server_negotiation.cc:407] Connection header received
0620 19:21:41.513954 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.513955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.513969 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.513996 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.514276 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33188: BlockingRecv error: recv got EOF from 127.0.0.1:33188 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 19:21:41.517136 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.516354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.1.1.253:36509)
0620 19:21:41.516382 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.516384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.516674 (+ 290us) server_negotiation.cc:407] Connection header received
0620 19:21:41.516757 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.516758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.516766 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.516792 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.517084 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33204: BlockingRecv error: recv got EOF from 127.0.0.1:33204 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:21:41.520833 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.520068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.1.1.253:36509)
0620 19:21:41.520100 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.520102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.520338 (+ 236us) server_negotiation.cc:407] Connection header received
0620 19:21:41.520431 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.520432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.520441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.520469 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.520784 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33220: BlockingRecv error: recv got EOF from 127.0.0.1:33220 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:21:41.526525 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.525791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.1.1.253:36509)
0620 19:21:41.525823 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.525824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.526056 (+ 232us) server_negotiation.cc:407] Connection header received
0620 19:21:41.526149 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.526150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.526159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.526187 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.526489 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33224: BlockingRecv error: recv got EOF from 127.0.0.1:33224 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:21:41.536216 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.535475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.1.1.253:36509)
0620 19:21:41.535508 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.535509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.535750 (+ 241us) server_negotiation.cc:407] Connection header received
0620 19:21:41.535844 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.535844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.535853 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.535881 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.536177 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33226: BlockingRecv error: recv got EOF from 127.0.0.1:33226 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:21:41.554016 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.553257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.1.1.253:36509)
0620 19:21:41.553302 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.553304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.553530 (+ 226us) server_negotiation.cc:407] Connection header received
0620 19:21:41.553620 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.553621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.553631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.553667 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.553975 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33236: BlockingRecv error: recv got EOF from 127.0.0.1:33236 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:41.587846 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.587172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.1.1.253:36509)
0620 19:21:41.587221 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.587223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.587330 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:21:41.587406 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.587407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.587422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.587483 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.587802 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33250: BlockingRecv error: recv got EOF from 127.0.0.1:33250 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:21:41.653386 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.652780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.1.1.253:36509)
0620 19:21:41.652836 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.652838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.652892 (+ 54us) server_negotiation.cc:407] Connection header received
0620 19:21:41.652939 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.652940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.652948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.653003 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.653343 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33260: BlockingRecv error: recv got EOF from 127.0.0.1:33260 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 19:21:41.783145 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:41.782450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.1.1.253:36509)
0620 19:21:41.782522 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:41.782524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:41.782629 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:21:41.782708 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:41.782710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:41.782725 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:41.782785 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:41.783085 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33262: BlockingRecv error: recv got EOF from 127.0.0.1:33262 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 19:21:42.040896 23404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:42.040294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.1.1.253:36509)
0620 19:21:42.040343 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.040345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.040374 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:21:42.040444 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.040445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.040455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.040505 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:42.040837 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33264: BlockingRecv error: recv got EOF from 127.0.0.1:33264 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:42.401122 23493 catalog_manager.cc:1259] Loaded cluster ID: a62d669f8d444e42a65c8e0fa4621716
I20260620 19:21:42.401196 23493 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: loading cluster ID for follower catalog manager: success
I20260620 19:21:42.401974 23493 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: acquiring CA information for follower catalog manager: success
I20260620 19:21:42.402275 23493 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 696698c6be2a4f68a0ebf85ca2eab25a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:21:42.418222 23507 catalog_manager.cc:1259] Loaded cluster ID: a62d669f8d444e42a65c8e0fa4621716
I20260620 19:21:42.418286 23507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: loading cluster ID for follower catalog manager: success
I20260620 19:21:42.418943 23507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: acquiring CA information for follower catalog manager: success
I20260620 19:21:42.419183 23507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0472e8f41e2046f7ae22b7596a632b3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:21:42.556602 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:42.553957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.1.1.253:36509)
0620 19:21:42.554068 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.554070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.554133 (+ 63us) server_negotiation.cc:407] Connection header received
0620 19:21:42.554224 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.554225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.554283 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.554366 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:42.554516 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.555020 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.555808 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.556003 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.556075 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:42.556239 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:42.556358 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:42.556440 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:42.556477 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:21:42.556497 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260620 19:21:42.560963 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:42.558792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.1.1.252:44533)
0620 19:21:42.558898 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.558900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.558907 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:42.558940 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.558941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.558973 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.559039 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:42.559116 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.559648 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.560299 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.560490 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.560538 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:42.560704 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:42.560792 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:42.560852 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:42.560889 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:21:42.560905 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
I20260620 19:21:42.565501 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:42.562651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.1.1.252:44533)
0620 19:21:42.562704 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.562706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.562777 (+ 71us) server_negotiation.cc:407] Connection header received
0620 19:21:42.562861 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.562861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.562871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.562934 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:42.563067 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.563577 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.564572 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.564849 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.564906 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:42.565116 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:42.565151 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:42.565205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:42.565394 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 19:21:42.565421 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:21:42.565723 23623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:42.562462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.1.1.254:41719)
0620 19:21:42.562570 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.562572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.562632 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:21:42.562717 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.562718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.562771 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.562849 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:42.562989 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.563780 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.564664 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.564976 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.565252 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:42.565312 (+ 60us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:42.565403 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:42.565464 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:42.565509 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:21:42.565531 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260620 19:21:42.565903 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:42.562459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.1.1.253:36509)
0620 19:21:42.562495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:42.562497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:42.562936 (+ 439us) server_negotiation.cc:407] Connection header received
0620 19:21:42.563021 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:42.563023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:42.563031 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:42.563098 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:42.563207 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.563897 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.564926 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:42.565228 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:42.565286 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:42.565458 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:42.565493 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:42.565537 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:42.565813 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 19:21:42.565841 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:42.567272 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23257
I20260620 19:21:42.571929 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23332
I20260620 19:21:42.576691 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23407
2026-06-20T19:21:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2703 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:21:42 dist-test-slave-mwkb krb5kdc[23627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:21:42 dist-test-slave-mwkb krb5kdc[23627](info): set up 2 sockets
Jun 20 19:21:42 dist-test-slave-mwkb krb5kdc[23627](info): commencing operation
krb5kdc: starting...
W20260620 19:21:43.577241 1031 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/test-user.keytab.
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.1.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
Entry for principal kudu/127.1.1.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab.
I20260620 19:21:43.683404 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41567
--webserver_interface=127.1.1.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:21:43.759528 23640 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:43.759773 23640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:43.759800 23640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:43.759841 23640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:43.760972 23640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:43.761034 23640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:43.761057 23640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:43.761076 23640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:43.761139 23640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:43.762607 23640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.254.keytab
--principal=kudu/127.1.1.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.254:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.1.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:43.762862 23640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:43.763070 23640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:43.767536 23645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:43.767529 23646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:43.767580 23648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:43.767638 23640 server_base.cc:1034] running on GCE node
I20260620 19:21:43.767994 23640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:43.768260 23640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:43.769411 23640 hybrid_clock.cc:648] HybridClock initialized: now 1781983303769394 us; error 35 us; skew 500 ppm
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:43.771528 23640 init.cc:377] Logged in from keytab as kudu/127.1.1.254@KRBTEST.COM (short username kudu)
I20260620 19:21:43.771889 23640 webserver.cc:460] Webserver started at http://127.1.1.254:37077/ using document root <none> and password file <none>
I20260620 19:21:43.772104 23640 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:43.772164 23640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:43.772301 23640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:43.773309 23640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data/instance:
uuid: "b8322ad1194d42b5afa34467b6af27f7"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "8629f3a8ae49f42df135c9b12a9402aa"
server_key_iv: "02bdfd45d68838c90ba6a959cf384f34"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.773666 23640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance:
uuid: "b8322ad1194d42b5afa34467b6af27f7"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "8629f3a8ae49f42df135c9b12a9402aa"
server_key_iv: "02bdfd45d68838c90ba6a959cf384f34"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.774829 23640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:21:43.775496 23655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:43.775646 23640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:21:43.775734 23640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal
uuid: "b8322ad1194d42b5afa34467b6af27f7"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "8629f3a8ae49f42df135c9b12a9402aa"
server_key_iv: "02bdfd45d68838c90ba6a959cf384f34"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.775805 23640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:43.785948 23640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:43.788156 23640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:43.788314 23640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:43.792220 23640 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.254:41567
I20260620 19:21:43.792263 23707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.254:41567 every 8 connection(s)
I20260620 19:21:43.792583 23640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/data/info.pb
I20260620 19:21:43.793869 23708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:43.795372 23708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:43.797281 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23640
I20260620 19:21:43.797384 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-0/wal/instance
I20260620 19:21:43.797564 1031 external_mini_cluster.cc:1428] Setting key ac03d9828463de07db1fe39b00be2880
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:43.804904 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.795997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41567 (local address 127.0.0.1:42594)
0620 19:21:43.796150 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.796186 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:43.796598 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:43.796840 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:43.796842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:43.797143 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:43.797296 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.797300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.798378 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.798379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.799034 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.799035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.799512 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.799513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.799577 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.800032 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:43.800042 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:43.801589 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:43.803594 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.803601 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.803602 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.803782 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.804043 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.804044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.804044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.804124 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.804522 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:43.804525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:43.804575 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:21:43.804647 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:21:43.804760 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260620 19:21:43.804997 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.796141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.1.1.254:41567)
0620 19:21:43.796249 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:43.796253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:43.796297 (+ 44us) server_negotiation.cc:407] Connection header received
0620 19:21:43.796724 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:43.796726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:43.796742 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:43.796817 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:43.797388 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.798279 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.799130 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.799412 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.799515 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.801674 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:43.801681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:43.801682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:43.801689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:43.803516 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.803860 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.803863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.803863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.803968 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.804183 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.804184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.804184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.804452 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:43.804545 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:43.804702 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 19:21:43.804839 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":77,"thread_start_us":52,"threads_started":1}
WARNING: no policy specified for kudu/127.1.1.253@KRBTEST.COM; defaulting to no policy
I20260620 19:21:43.805655 23708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:43.805976 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.805853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38048)
0620 19:21:43.805900 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.805947 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:21:43.806110 23659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
I20260620 19:21:43.806449 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.806397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38058)
0620 19:21:43.806418 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.806437 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 19:21:43.806551 23708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:32915: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
Entry for principal kudu/127.1.1.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab.
I20260620 19:21:43.820848 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:32915
--webserver_interface=127.1.1.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:21:43.855789 23708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } attempt: 1
I20260620 19:21:43.856179 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.856033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38062)
0620 19:21:43.856092 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.856147 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:21:43.856515 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.856420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38070)
0620 19:21:43.856473 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.856499 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 19:21:43.856660 23708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:32915: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
W20260620 19:21:43.895526 23715 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:43.895717 23715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:43.895735 23715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:43.895756 23715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:43.896919 23715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:43.896958 23715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:43.896973 23715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:43.896986 23715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:43.896997 23715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:43.898448 23715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.253.keytab
--principal=kudu/127.1.1.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.253:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.1.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:43.898682 23715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:43.898895 23715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:43.903394 23723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:43.903429 23721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:43.903394 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:43.903493 23715 server_base.cc:1034] running on GCE node
I20260620 19:21:43.903805 23715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:43.904028 23715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:43.905205 23715 hybrid_clock.cc:648] HybridClock initialized: now 1781983303905187 us; error 32 us; skew 500 ppm
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:43.907519 23715 init.cc:377] Logged in from keytab as kudu/127.1.1.253@KRBTEST.COM (short username kudu)
I20260620 19:21:43.907951 23715 webserver.cc:460] Webserver started at http://127.1.1.253:40467/ using document root <none> and password file <none>
I20260620 19:21:43.908143 23715 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:43.908176 23715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:43.908246 23715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:43.909137 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data/instance:
uuid: "f8982aa8e5bf45dc84b0eabb98135d26"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "5cb6f25b2d9a95e8da0a83d6582be67e"
server_key_iv: "b46668461fa63fb162485fd6a95188e9"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.909401 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal/instance:
uuid: "f8982aa8e5bf45dc84b0eabb98135d26"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "5cb6f25b2d9a95e8da0a83d6582be67e"
server_key_iv: "b46668461fa63fb162485fd6a95188e9"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.910486 23715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:21:43.911094 23730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:43.911258 23715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:21:43.911338 23715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal
uuid: "f8982aa8e5bf45dc84b0eabb98135d26"
format_stamp: "Formatted at 2026-06-20 19:21:43 on dist-test-slave-mwkb"
server_key: "5cb6f25b2d9a95e8da0a83d6582be67e"
server_key_iv: "b46668461fa63fb162485fd6a95188e9"
server_key_version: "encryptionkey@0"
I20260620 19:21:43.911383 23715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:43.924844 23708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } attempt: 2
I20260620 19:21:43.925195 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.925063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38076)
0620 19:21:43.925132 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.925162 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 19:21:43.925491 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.925441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38086)
0620 19:21:43.925461 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.925480 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 19:21:43.925576 23708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.253:32915: Network error: Client connection negotiation failed: client connection to 127.1.1.253:32915: connect: Connection refused (error 111)
I20260620 19:21:43.935741 23715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:43.937440 23715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:43.937628 23715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:43.941305 23715 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.253:32915
I20260620 19:21:43.941349 23782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.253:32915 every 8 connection(s)
I20260620 19:21:43.941646 23715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/data/info.pb
I20260620 19:21:43.942516 23783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:43.943805 23783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:43.944744 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23715
I20260620 19:21:43.944837 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-1/wal/instance
I20260620 19:21:43.944994 1031 external_mini_cluster.cc:1428] Setting key 769cd87107b0bfc2f020a9fc7201cc54
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.1.252@KRBTEST.COM; defaulting to no policy
I20260620 19:21:43.952764 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.944358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.1.1.254:41567)
0620 19:21:43.944411 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:43.944413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:43.944586 (+ 173us) server_negotiation.cc:407] Connection header received
0620 19:21:43.945044 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:43.945046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:43.945058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:43.945142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:43.945727 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.946445 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.947837 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.948003 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.948054 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.950257 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:43.950259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:43.950259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:43.950266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:43.951640 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.951975 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.951976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.951976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.952029 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.952220 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.952221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.952221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.952340 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:43.952401 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:43.952593 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:21:43.952679 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:43.952819 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.944333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41567 (local address 127.0.0.1:42606)
0620 19:21:43.944482 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.944517 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:43.944963 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:43.945200 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:43.945203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:43.945468 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:43.945643 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.945647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.946541 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.946542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.947759 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.947762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.948117 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.948118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.948274 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.948747 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:43.948765 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:43.950178 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:43.951710 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.951715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.951717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.951911 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.952089 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.952089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.952089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.952164 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.952403 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:43.952406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:43.952492 (+ 86us) client_negotiation.cc:769] Sending connection context
0620 19:21:43.952552 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:21:43.952676 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260620 19:21:43.953281 23783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.1.252@KRBTEST.COM" created.
Jun 20 19:21:43 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:43.960469 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.953480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38094)
0620 19:21:43.953530 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.953539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:43.953574 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:43.953754 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:43.953755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:43.953885 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:43.953920 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.953921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.954792 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.954793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.955460 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:43.955463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.955905 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.955906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:43.955990 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.956393 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:43.956398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:43.957524 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:43.959521 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.959521 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.959522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.959694 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.959913 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:43.959913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:43.959913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:43.959959 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:43.960262 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:43.960263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:43.960293 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 19:21:43.960342 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:21:43.960411 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:21:43.960597 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.953524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.1.1.253:32915)
0620 19:21:43.953636 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:43.953639 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:43.953656 (+ 17us) server_negotiation.cc:407] Connection header received
0620 19:21:43.953671 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:43.953673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:43.953687 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:43.953754 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:43.953990 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.954706 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.955550 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:43.955837 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:43.955889 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:43.957598 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:43.957605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:43.957606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:43.957612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:43.959439 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.959771 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.959775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.959775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.959840 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:43.960017 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:43.960018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:43.960018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:43.960198 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:43.960284 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:43.960376 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:21:43.960480 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260620 19:21:43.961148 23783 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:43.961453 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.961336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48160)
0620 19:21:43.961380 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.961428 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 19:21:43.961593 23733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
I20260620 19:21:43.962033 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:43.961918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48170)
0620 19:21:43.961972 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:43.962003 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:21:43.962153 23783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:45245: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
Entry for principal kudu/127.1.1.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab.
I20260620 19:21:43.969710 1031 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNtb4aU/build/release/bin/kudu
/tmp/dist-test-taskNtb4aU/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:45245
--webserver_interface=127.1.1.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.1.212:35707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:21:44.011377 23783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } attempt: 1
I20260620 19:21:44.011828 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.011683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48172)
0620 19:21:44.011747 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.011791 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:21:44.012180 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.012089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48182)
0620 19:21:44.012133 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.012159 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:21:44.012295 23783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:45245: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
I20260620 19:21:44.016703 23708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } attempt: 3
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:44.022923 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.017001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.1.1.253:32915)
0620 19:21:44.017054 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.017056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.017065 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:44.017192 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.017192 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.017204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.017261 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.017544 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.017995 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.018644 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.018837 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.018915 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.020642 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.020643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.020644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.020651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.021944 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.022250 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.022251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.022251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.022303 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.022475 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.022476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.022477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.022572 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.022647 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.022776 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.022848 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:44.022928 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.016953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38098)
0620 19:21:44.016993 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.017007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.017064 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.017266 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.017267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.017410 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.017466 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.017467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.018074 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.018075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.018548 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.018549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.018915 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.018916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.018986 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.019394 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.019398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.020556 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.022031 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.022032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.022033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.022186 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.022362 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.022363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.022363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.022414 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.022638 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.022638 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.022679 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.022754 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.022842 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 19:21:44.023279 23708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:21:44.023572 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.023476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48198)
0620 19:21:44.023521 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.023550 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:21:44.023851 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.023768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48214)
0620 19:21:44.023809 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.023835 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:21:44.023945 23708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:45245: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
W20260620 19:21:44.045748 23790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:21:44.045922 23790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:21:44.045962 23790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:21:44.046003 23790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:21:44.047030 23790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:21:44.047065 23790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:21:44.047077 23790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:21:44.047120 23790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:21:44.047163 23790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:21:44.048367 23790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.1.212:35707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.1.254:41567,127.1.1.253:32915,127.1.1.252:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/krb5kdc/kudu_127.1.1.252.keytab
--principal=kudu/127.1.1.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.1.252:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.1.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:21:44.048560 23790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:21:44.048744 23790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:21:44.052995 23795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:44.053014 23796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:21:44.052981 23798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:21:44.053046 23790 server_base.cc:1034] running on GCE node
I20260620 19:21:44.053412 23790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:21:44.053681 23790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:21:44.054816 23790 hybrid_clock.cc:648] HybridClock initialized: now 1781983304054806 us; error 32 us; skew 500 ppm
I20260620 19:21:44.055055 23708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } attempt: 1
I20260620 19:21:44.055326 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.055227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48216)
0620 19:21:44.055267 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.055291 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:21:44.055655 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.055566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48230)
0620 19:21:44.055612 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.055638 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 19:21:44.055758 23708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:45245: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:21:44.057245 23790 init.cc:377] Logged in from keytab as kudu/127.1.1.252@KRBTEST.COM (short username kudu)
I20260620 19:21:44.057679 23790 webserver.cc:460] Webserver started at http://127.1.1.252:40579/ using document root <none> and password file <none>
I20260620 19:21:44.057884 23790 fs_manager.cc:359] Metadata directory not provided
I20260620 19:21:44.057929 23790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:21:44.058043 23790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:21:44.058988 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data/instance:
uuid: "398f6ee832754b79aa5c119a141d45a7"
format_stamp: "Formatted at 2026-06-20 19:21:44 on dist-test-slave-mwkb"
server_key: "54f15af9929066fae8d3a0737fd4be10"
server_key_iv: "46f43922609f8226afc966f74ebd8cb5"
server_key_version: "encryptionkey@0"
I20260620 19:21:44.059324 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal/instance:
uuid: "398f6ee832754b79aa5c119a141d45a7"
format_stamp: "Formatted at 2026-06-20 19:21:44 on dist-test-slave-mwkb"
server_key: "54f15af9929066fae8d3a0737fd4be10"
server_key_iv: "46f43922609f8226afc966f74ebd8cb5"
server_key_version: "encryptionkey@0"
I20260620 19:21:44.060585 23790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:21:44.061352 23805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:21:44.061512 23790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:21:44.061605 23790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data,/tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal
uuid: "398f6ee832754b79aa5c119a141d45a7"
format_stamp: "Formatted at 2026-06-20 19:21:44 on dist-test-slave-mwkb"
server_key: "54f15af9929066fae8d3a0737fd4be10"
server_key_iv: "46f43922609f8226afc966f74ebd8cb5"
server_key_version: "encryptionkey@0"
I20260620 19:21:44.061682 23790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:21:44.077373 23790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:21:44.079334 23790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:21:44.079480 23790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:21:44.080435 23783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } attempt: 2
I20260620 19:21:44.080740 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.080640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48242)
0620 19:21:44.080688 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.080720 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:21:44.081064 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.080984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48258)
0620 19:21:44.081022 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.081044 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:21:44.081167 23783 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.1.252:45245: Network error: Client connection negotiation failed: client connection to 127.1.1.252:45245: connect: Connection refused (error 111)
I20260620 19:21:44.083688 23790 rpc_server.cc:307] RPC server started. Bound to: 127.1.1.252:45245
I20260620 19:21:44.083719 23857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.1.252:45245 every 8 connection(s)
I20260620 19:21:44.084035 23790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/data/info.pb
I20260620 19:21:44.084925 23858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:21:44.086336 23858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
I20260620 19:21:44.093547 1031 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNtb4aU/build/release/bin/kudu as pid 23790
I20260620 19:21:44.093654 1031 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNtb4aU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781983021660275-1031-0/minicluster-data/master-2/wal/instance
I20260620 19:21:44.093863 1031 external_mini_cluster.cc:1428] Setting key 7edb70d3b8ba4cd0c2f98a5955fe943a
I20260620 19:21:44.097337 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.086940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.1.1.254:41567)
0620 19:21:44.086985 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.086986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.087176 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:21:44.087575 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.087576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.087661 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.087730 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.088318 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.088781 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.089994 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.090189 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.090255 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.092545 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.092549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.092549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.092560 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.095348 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.095963 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.095964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.095965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.096032 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.096618 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.096619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.096619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.096737 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.096801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.097200 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.097267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:21:44.097558 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.086878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.254:41567 (local address 127.0.0.1:42622)
0620 19:21:44.087035 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.087074 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.087497 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.087827 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.087830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.088053 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.088234 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.088237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.088952 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.088953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.089898 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.089900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.090309 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.090310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.090416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.090860 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.090867 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.092388 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.095508 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.095528 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.095530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.095876 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.096204 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.096205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.096206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.096374 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.096816 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.096823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.097000 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.097155 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.097327 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260620 19:21:44.098248 23858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:44.104341 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.094272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.1.1.252:45245)
0620 19:21:44.094426 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.094436 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.094622 (+ 186us) server_negotiation.cc:407] Connection header received
0620 19:21:44.094659 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.094661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.094681 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.094767 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.095551 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.096363 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.097552 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.098054 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.098114 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.101165 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.101180 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.101180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.101186 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.102981 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.103391 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.103394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.103395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.103453 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.103681 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.103682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.103683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.103854 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.104001 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.104140 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.104219 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260620 19:21:44.106312 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.098497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.253:32915 (local address 127.0.0.1:38110)
0620 19:21:44.098546 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.098556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.098594 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.098748 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.098749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.098868 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.098998 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.099001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.099979 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.099980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.100743 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.100745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.101252 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.101253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.101332 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.101784 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.101789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.103185 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.105236 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.105238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.105238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.105476 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.105675 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.105675 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.105676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.105733 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.106003 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.106004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.106043 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.106121 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.106216 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:21:44.106313 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.098578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.1.1.253:32915)
0620 19:21:44.098631 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.098633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.098641 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:44.098663 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.098664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.098674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.098732 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.099099 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.099875 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.100824 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.101188 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.101252 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.103273 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.103274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.103274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.103280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.105132 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.105538 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.105539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.105539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.105597 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.105814 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.105815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.105816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.105927 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.105997 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.106160 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.106231 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:21:44.106668 23858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.254@KRBTEST.COM
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983304, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.252@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:21:44.114071 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.106944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48278)
0620 19:21:44.107008 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.107018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.107060 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.107216 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.107217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.107321 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.107348 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.107349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.108136 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.108137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.108681 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.108683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.109053 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.109054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.109123 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.109514 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.109518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.111166 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.112878 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.112878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.112879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.113060 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.113301 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.113301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.113302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.113389 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.113669 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.113670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.113716 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.113887 (+ 171us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.113983 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 19:21:44.114068 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.105393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.1.1.254:41567)
0620 19:21:44.105426 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.105427 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.105435 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:44.105496 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.105496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.105504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.105571 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.106116 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.107024 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.107969 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.108189 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.108264 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.110763 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.110765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.110765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.110773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.112698 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.113057 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.113058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.113059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.113137 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.113347 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.113348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.113349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.113505 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.113751 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.113897 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.113977 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:21:44.114182 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.106978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.1.1.252:45245)
0620 19:21:44.107007 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.107008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.107048 (+ 40us) server_negotiation.cc:407] Connection header received
0620 19:21:44.107127 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.107128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.107137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.107214 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.107414 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.108069 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.108765 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.108970 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.109052 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.111257 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.111258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.111259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.111265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.112798 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.113181 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.113182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.113183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.113242 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.113454 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.113454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.113454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.113544 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.113685 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.113932 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.114042 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":15}
I20260620 19:21:44.115929 23858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: Bootstrap starting.
I20260620 19:21:44.116971 23858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:44.117365 23858 log.cc:824] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:44.118214 23858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: No bootstrap required, opened a new log
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.1.253@KRBTEST.COM
I20260620 19:21:44.120151 23858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.120311 23858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398f6ee832754b79aa5c119a141d45a7, State: Initialized, Role: FOLLOWER
I20260620 19:21:44.120396 23858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.120725 23858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:44.120694 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.120908 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.122676 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.114594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38118 (local address 127.1.1.253:32915)
0620 19:21:44.114637 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.114639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.114932 (+ 293us) server_negotiation.cc:407] Connection header received
0620 19:21:44.114953 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.114954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.114964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.115014 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.115447 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.116275 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.116918 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.117324 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.117405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.119680 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.119681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.119682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.119688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.121254 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.121651 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.121652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.121653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.121708 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.121912 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.121913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.121914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.122055 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.122215 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.122505 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.122575 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 19:21:44.122632 23875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:44.122802 23875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:44.123826 1031 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:21:44.124574 23682 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b8322ad1194d42b5afa34467b6af27f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42634
I20260620 19:21:44.123314 23790 master_runner.cc:186] Error getting master registration for 127.1.1.254:41567: OK, instance_id { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" instance_seqno: 1781983303791190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:21:44.126035 23790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:21:44.130882 23708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } attempt: 2
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.254@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:21:44.138123 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.131218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.1.1.252:45245)
0620 19:21:44.131271 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.131273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.131281 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:21:44.131298 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.131299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.131308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.131361 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.131627 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.132363 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.133279 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.133587 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.133653 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.135553 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.135555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.135555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.135561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.137029 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.137398 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.137398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.137399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.137444 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.137616 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.137617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.137617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.137744 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.137799 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.137967 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.138027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:21:44.138136 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.131140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48286)
0620 19:21:44.131195 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.131206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.131247 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.131382 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.131383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.131525 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.131561 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.131562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.132440 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.132441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.133195 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.133196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.133661 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.133662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.133712 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.134121 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.134125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.135467 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.137131 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.137131 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.137132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.137331 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.137505 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.137506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.137506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.137562 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.137808 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.137808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.137864 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.137940 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.138020 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:21:44.139531 23708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: Bootstrap starting.
I20260620 19:21:44.140231 23708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:44.140501 23708 log.cc:824] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:44.141310 23708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: No bootstrap required, opened a new log
I20260620 19:21:44.142566 23708 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.142683 23708 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8322ad1194d42b5afa34467b6af27f7, State: Initialized, Role: FOLLOWER
I20260620 19:21:44.142763 23708 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.143005 23878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.143071 23708 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:21:44.143095 23878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.143189 23682 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:21:44.143289 23682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:44.144016 23682 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.144744 23682 leader_election.cc:288] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [CANDIDATE]: Term 1 election: Requested vote from peers f8982aa8e5bf45dc84b0eabb98135d26 (127.1.1.253:32915), 398f6ee832754b79aa5c119a141d45a7 (127.1.1.252:45245)
W20260620 19:21:44.144845 23889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:44.144898 23889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:44.145198 23832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8322ad1194d42b5afa34467b6af27f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "398f6ee832754b79aa5c119a141d45a7"
I20260620 19:21:44.145310 23832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:44.146163 23832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8322ad1194d42b5afa34467b6af27f7 in term 1.
I20260620 19:21:44.145553 23640 master_runner.cc:186] Error getting master registration for 127.1.1.253:32915: OK, instance_id { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" instance_seqno: 1781983303940258 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:21:44.147816 23640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:21:44.147987 23658 leader_election.cc:302] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398f6ee832754b79aa5c119a141d45a7, b8322ad1194d42b5afa34467b6af27f7; no voters:
I20260620 19:21:44.148094 23878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:21:44.148243 23878 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [term 1 LEADER]: Becoming Leader. State: Replica: b8322ad1194d42b5afa34467b6af27f7, State: Running, Role: LEADER
I20260620 19:21:44.148312 23878 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.148682 23878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8322ad1194d42b5afa34467b6af27f7. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.148762 23878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:44.149215 23900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:21:44.149426 23900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:21:44.161965 23832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [term 1 FOLLOWER]: Refusing update from remote peer b8322ad1194d42b5afa34467b6af27f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:44.162282 23878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:44.163566 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.149056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.1.1.254:41567)
0620 19:21:44.149114 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.149116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.152688 (+ 3572us) server_negotiation.cc:407] Connection header received
0620 19:21:44.153051 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.153052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.153064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.153142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.153806 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.154504 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.155603 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.155896 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.155935 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.159349 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.159351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.159352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.159359 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.161444 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.161958 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.161959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.161959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.162029 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.162356 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.162357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.162357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.162489 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.162574 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.163379 (+ 805us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.163459 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:21:44.163848 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8322ad1194d42b5afa34467b6af27f7. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.163959 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.165129 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.165217 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.165238 23878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.165311 23878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:44.165591 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.147686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.1.1.252:45245)
0620 19:21:44.147715 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.147717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.149272 (+ 1555us) server_negotiation.cc:407] Connection header received
0620 19:21:44.149520 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.149521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.149529 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.149568 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.153096 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.153729 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.157146 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.157430 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.157471 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.159883 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.159884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.159885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.159902 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.161540 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.164417 (+ 2877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.164418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.164419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.164491 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.164838 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.164839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.164839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.164974 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.165041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.165437 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.165516 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:21:44.165792 23902 mvcc.cc:205] Tried to move back new op lower bound from 7299003613844852736 to 7299003613792063488. Current Snapshot: MvccSnapshot[applied={T|T < 7299003613844852736}]
I20260620 19:21:44.167128 23787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.147286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.1.1.253:32915)
0620 19:21:44.147322 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.147323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.148354 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 19:21:44.148467 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.148468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.148487 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.148525 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.150330 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.151060 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.154982 (+ 3922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.155351 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.155398 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.161796 (+ 6398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.161798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.161798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.161805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.163581 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.166135 (+ 2554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.166136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.166137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.166194 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.166451 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.166451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.166452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.166570 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.166662 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.166932 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.167036 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:21:44.169236 23900 catalog_manager.cc:1347] Generated new cluster ID: 5e09b086220e44479cb3a8392f8aab5a
I20260620 19:21:44.169283 23900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:21:44.172298 23783 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } attempt: 3
Jun 20 19:21:44 dist-test-slave-mwkb krb5kdc[23627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983303, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.1.253@KRBTEST.COM for kudu/127.1.1.252@KRBTEST.COM
I20260620 19:21:44.179968 23900 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:21:44.180631 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.172552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.1.252:45245 (local address 127.0.0.1:48300)
0620 19:21:44.172651 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:21:44.172659 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:21:44.172714 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:21:44.172879 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:21:44.172879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:21:44.173028 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:21:44.173063 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.173064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.173895 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.173896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.174551 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:21:44.174552 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.174955 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.174955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:21:44.174995 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.175321 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:21:44.175325 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:21:44.177502 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:21:44.179612 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.179613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.179613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.179830 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.180019 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:21:44.180020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:21:44.180020 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:21:44.180076 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:21:44.180339 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:21:44.180340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:21:44.180393 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 19:21:44.180457 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:21:44.180539 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 19:21:44.181133 23900 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:21:44.181293 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:44.172581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.1.1.252:45245)
0620 19:21:44.172614 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.172615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.172769 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:21:44.172787 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.172788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.172796 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.172875 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.173158 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.173825 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.174614 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:44.174863 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:44.174903 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:44.177585 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:21:44.177586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:21:44.177587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:21:44.177593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:21:44.179504 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.179895 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.179896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.179897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.179953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:21:44.180154 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:21:44.180155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:21:44.180155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:21:44.180288 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:21:44.180348 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:44.181029 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 19:21:44.181129 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 19:21:44.182623 23783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: Bootstrap starting.
I20260620 19:21:44.183193 23783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: Neither blocks nor log segments found. Creating new log.
I20260620 19:21:44.183414 23783 log.cc:824] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: Log is configured to *not* fsync() on all Append() calls
I20260620 19:21:44.184460 23783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: No bootstrap required, opened a new log
I20260620 19:21:44.186473 23783 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.186589 23783 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8982aa8e5bf45dc84b0eabb98135d26, State: Initialized, Role: FOLLOWER
I20260620 19:21:44.186650 23783 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } }
I20260620 19:21:44.186925 23783 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:21:44.188306 23736 tablet.cc:2369] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:21:44.188482 23757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8322ad1194d42b5afa34467b6af27f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f8982aa8e5bf45dc84b0eabb98135d26"
I20260620 19:21:44.188831 23756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:21:44.190776 23756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [term 1 FOLLOWER]: Refusing update from remote peer b8322ad1194d42b5afa34467b6af27f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:21:44.191067 23904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:21:44.191941 23757 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8322ad1194d42b5afa34467b6af27f7 in term 1.
I20260620 19:21:44.195034 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.195138 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.195196 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8322ad1194d42b5afa34467b6af27f7. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.195259 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:21:44.195309 23908 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.195370 23908 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:21:44.195875 23923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:21:44.196038 23923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:21:44.196939 23900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7: Generated new TSK 0
I20260620 19:21:44.197183 23900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:21:44.197423 23904 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8322ad1194d42b5afa34467b6af27f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8322ad1194d42b5afa34467b6af27f7" member_type: VOTER last_known_addr { host: "127.1.1.254" port: 41567 } } peers { permanent_uuid: "f8982aa8e5bf45dc84b0eabb98135d26" member_type: VOTER last_known_addr { host: "127.1.1.253" port: 32915 } } peers { permanent_uuid: "398f6ee832754b79aa5c119a141d45a7" member_type: VOTER last_known_addr { host: "127.1.1.252" port: 45245 } } }
I20260620 19:21:44.197539 23904 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8322ad1194d42b5afa34467b6af27f7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:21:44.197779 23915 mvcc.cc:205] Tried to move back new op lower bound from 7299003613911183360 to 7299003613792063488. Current Snapshot: MvccSnapshot[applied={T|T < 7299003613844852736}]
W20260620 19:21:44.226783 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.225298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.1.1.253:32915)
0620 19:21:44.225354 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.225356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.226180 (+ 824us) server_negotiation.cc:407] Connection header received
0620 19:21:44.226275 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.226276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.226287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.226316 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.226715 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38136: BlockingRecv error: recv got EOF from 127.0.0.1:38136 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:44.227665 23862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.225647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.1.1.252:45245)
0620 19:21:44.225682 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.225683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.227217 (+ 1534us) server_negotiation.cc:407] Connection header received
0620 19:21:44.227303 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.227304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.227313 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.227341 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.227612 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48310: BlockingRecv error: recv got EOF from 127.0.0.1:48310 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:44.228461 23711 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.225943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.1.1.254:41567)
0620 19:21:44.225979 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.225981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.227986 (+ 2005us) server_negotiation.cc:407] Connection header received
0620 19:21:44.228102 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.228103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.228112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.228139 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.228406 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42650: BlockingRecv error: recv got EOF from 127.0.0.1:42650 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:44.230581 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.229935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.1.1.253:32915)
0620 19:21:44.229964 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.229965 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.230162 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:21:44.230249 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.230250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.230263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.230287 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.230545 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38148: BlockingRecv error: recv got EOF from 127.0.0.1:38148 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:21:44.233296 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.232616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.1.1.253:32915)
0620 19:21:44.232644 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.232646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.232860 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:21:44.232938 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.232939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.232947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.232971 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.233259 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38158: BlockingRecv error: recv got EOF from 127.0.0.1:38158 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:21:44.236950 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.236294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.1.1.253:32915)
0620 19:21:44.236334 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.236336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.236568 (+ 232us) server_negotiation.cc:407] Connection header received
0620 19:21:44.236641 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.236642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.236649 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.236673 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.236918 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38170: BlockingRecv error: recv got EOF from 127.0.0.1:38170 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:21:44.242432 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.241844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.1.1.253:32915)
0620 19:21:44.241871 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.241872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.242059 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:21:44.242132 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.242132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.242139 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.242162 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.242402 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38174: BlockingRecv error: recv got EOF from 127.0.0.1:38174 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:21:44.251973 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.251306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.1.1.253:32915)
0620 19:21:44.251334 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.251335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.251566 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:21:44.251644 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.251645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.251653 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.251677 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.251939 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 19:21:44.270028 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.269166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.1.1.253:32915)
0620 19:21:44.269205 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.269207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.269476 (+ 269us) server_negotiation.cc:407] Connection header received
0620 19:21:44.269580 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.269582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.269592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.269622 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.269980 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38180: BlockingRecv error: recv got EOF from 127.0.0.1:38180 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 19:21:44.304298 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.303373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.1.1.253:32915)
0620 19:21:44.303417 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.303419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.303697 (+ 278us) server_negotiation.cc:407] Connection header received
0620 19:21:44.303801 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.303802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.303814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.303844 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.304230 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38184: BlockingRecv error: recv got EOF from 127.0.0.1:38184 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:21:44.370425 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.369734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.1.1.253:32915)
0620 19:21:44.369793 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.369795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.369920 (+ 125us) server_negotiation.cc:407] Connection header received
0620 19:21:44.369958 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.369959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.369968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.370022 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.370373 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38188: BlockingRecv error: recv got EOF from 127.0.0.1:38188 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 19:21:44.500211 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.499669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.1.1.253:32915)
0620 19:21:44.499738 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.499740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.499749 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:44.499773 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.499774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.499783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.499848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.500137 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38194: BlockingRecv error: recv got EOF from 127.0.0.1:38194 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 19:21:44.758275 23787 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:21:44.757610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.1.1.253:32915)
0620 19:21:44.757674 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:44.757676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:44.757685 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:21:44.757781 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:44.757782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:44.757794 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:44.757866 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:21:44.758202 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38206: BlockingRecv error: recv got EOF from 127.0.0.1:38206 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:21:45.123214 23875 catalog_manager.cc:1259] Loaded cluster ID: 5e09b086220e44479cb3a8392f8aab5a
I20260620 19:21:45.123286 23875 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: loading cluster ID for follower catalog manager: success
I20260620 19:21:45.123996 23875 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: acquiring CA information for follower catalog manager: success
I20260620 19:21:45.124269 23875 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 398f6ee832754b79aa5c119a141d45a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:21:45.196972 23923 catalog_manager.cc:1259] Loaded cluster ID: 5e09b086220e44479cb3a8392f8aab5a
I20260620 19:21:45.197046 23923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: loading cluster ID for follower catalog manager: success
I20260620 19:21:45.197917 23923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: acquiring CA information for follower catalog manager: success
I20260620 19:21:45.198210 23923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8982aa8e5bf45dc84b0eabb98135d26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:21:45.273540 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:45.271371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.1.1.253:32915)
0620 19:21:45.271469 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:45.271470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:45.271536 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:21:45.271583 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:45.271584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:45.271631 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:45.271691 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:45.271821 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.272268 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.272903 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.273057 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.273153 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:45.273269 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:45.273358 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:45.273407 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:45.273436 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 19:21:45.273451 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260620 19:21:45.277627 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:45.275096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.1.1.252:45245)
0620 19:21:45.275188 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:45.275190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:45.275197 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:21:45.275236 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:45.275237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:45.275265 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:45.275318 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:45.275391 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.275912 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.276947 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.277212 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.277263 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:45.277368 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:45.277436 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:45.277476 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:45.277498 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 19:21:45.277513 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":56,"threads_started":1}
I20260620 19:21:45.281796 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:45.279067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.1.1.253:32915)
0620 19:21:45.279313 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:45.279315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:45.279325 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:21:45.279347 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:45.279348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:45.279357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:45.279418 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:45.279508 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.280216 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281161 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.281366 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281429 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:45.281554 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:45.281577 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:45.281601 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:45.281704 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 19:21:45.281727 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260620 19:21:45.281931 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:45.279251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.1.1.252:45245)
0620 19:21:45.279319 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:45.279321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:45.279518 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:21:45.279583 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:45.279584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:45.279605 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:45.279663 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:45.279812 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.280304 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281211 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.281447 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281498 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:45.281667 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:45.281719 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:45.281759 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:45.281840 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:21:45.281876 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:21:45.282413 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:21:45.279073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.1.1.254:41567)
0620 19:21:45.279281 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:21:45.279283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:21:45.279395 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:21:45.279478 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:21:45.279479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:21:45.279523 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:21:45.279585 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:21:45.279862 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.280611 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281520 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:21:45.281800 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:21:45.281848 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:21:45.282028 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:21:45.282116 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:21:45.282174 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:21:45.282316 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:21:45.282340 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":46,"threads_started":1}
I20260620 19:21:45.283838 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23640
I20260620 19:21:45.290077 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23715
I20260620 19:21:45.295851 1031 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNtb4aU/build/release/bin/kudu with pid 23790
2026-06-20T19:21:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2719 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5423 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283643 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS